Compare commits
18 Commits
v0.1.8
...
9adfa91e74
| Author | SHA1 | Date | |
|---|---|---|---|
| 9adfa91e74 | |||
| 65d8ed80b5 | |||
| cffe161f29 | |||
| 5248e7de4a | |||
| d5d0df792b | |||
| 15ade8df75 | |||
| f3c485ef61 | |||
| 0e36b20a14 | |||
| 06e6e7191b | |||
| 3449cbc8a5 | |||
| 92950af6f4 | |||
| 8826cf2890 | |||
| f2dfb6c90e | |||
| 3bc1d3489e | |||
| d1051293b2 | |||
| 8c5f149f07 | |||
| f3210234d3 | |||
| b8b395e8b5 |
32
README.md
32
README.md
@@ -78,7 +78,17 @@ Profiles can be supplied as JSON through
|
||||
`GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON`, or from a JSON file path through
|
||||
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. Each profile has an `id`, `provider`,
|
||||
`endpoint_url`, governance `scope_type`/`scope_id`, optional `capabilities`, and
|
||||
credential references such as `password_env`, `token_env`, or `secret_ref`.
|
||||
deployment-owned credential references such as `password_env`, `token_env`, or
|
||||
`secret_ref`. Environment references require an exact name in the deployment-wide
|
||||
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; API-managed profiles cannot select
|
||||
process environment variables and may use only Files-owned encrypted password or
|
||||
token values. API-created `secret_ref` values fail closed until Files has an
|
||||
ownership contract that can confirm provider-side deletion. Legacy external
|
||||
references are treated as non-owned: deleting a profile or credential detaches
|
||||
and audits the reference but never passes it to an arbitrary secret provider.
|
||||
Profile and credential deletion immediately clears encrypted values, credential
|
||||
identities, deployment references, and private metadata in the same transaction
|
||||
as the non-secret audit record; an audit failure rolls the deletion back.
|
||||
`GET /api/v1/files/connectors/profiles` returns only profiles visible to the
|
||||
current principal (system, tenant, user, group, or accessible campaign scope) and
|
||||
redacts secret values and environment variable names. Use the returned
|
||||
@@ -102,8 +112,24 @@ conflict handling, source provenance, and connector audit path as direct
|
||||
uploads. The Seafile provider uses account-token auth and the native file
|
||||
download-link API; Nextcloud and generic WebDAV profiles use authenticated `GET`
|
||||
requests against the configured WebDAV endpoint. SMB profiles use
|
||||
`smb://server[:port]/share[/path]` endpoints and environment-backed credentials
|
||||
through `smbprotocol`.
|
||||
`smb://server[:port]/share[/path]` endpoints and deployment-owned or encrypted
|
||||
stored credentials through `smbprotocol`. Because that SDK cannot accept a
|
||||
preconnected socket and may follow DFS referrals to additional hosts, live SMB
|
||||
access fails closed in both public-only and private-network deployments. It will
|
||||
remain disabled until every initial connection and referral target can be
|
||||
policy-validated and pinned.
|
||||
|
||||
The S3 browse/import implementation and S3 managed-storage backend currently
|
||||
fail closed before creating a `boto3` client. Botocore does not yet use the
|
||||
GovOPlaN pinned HTTP transport and may manage redirects itself, so neither an
|
||||
explicit endpoint nor SDK endpoint discovery is allowed until both peer pinning
|
||||
and redirect revalidation are enforced.
|
||||
|
||||
Destructive Files-module retirement applies the same credential lifecycle before
|
||||
dropping tables. Every remaining Files-owned encrypted connector secret is
|
||||
scrubbed and audited first, while legacy non-owned external references are
|
||||
detached and identified as such in the audit record. Retirement does not claim
|
||||
or attempt provider-side deletion for references Files cannot prove it owns.
|
||||
|
||||
Local connector development assets live in `dev/connectors/`. The compose stack
|
||||
boots Nextcloud, Seafile, WebDAV, and SMB endpoints for provider development and
|
||||
|
||||
@@ -17,3 +17,9 @@ SMB_HOST_PORT=1445
|
||||
SMB_SHARE_NAME=files
|
||||
SMB_USER=govoplan
|
||||
SMB_PASSWORD=govoplan-smb
|
||||
|
||||
MINIO_API_HOST_PORT=9000
|
||||
MINIO_CONSOLE_HOST_PORT=9001
|
||||
MINIO_ROOT_USER=govoplan
|
||||
MINIO_ROOT_PASSWORD=govoplan-minio
|
||||
MINIO_BUCKET=govoplan
|
||||
|
||||
@@ -9,7 +9,8 @@ binds services to localhost high ports.
|
||||
```bash
|
||||
cd /mnt/DATA/git/govoplan-files/dev/connectors
|
||||
cp .env.example .env
|
||||
docker compose up -d nextcloud nextcloud-db webdav smb
|
||||
mkdir -p data/smb data/webdav
|
||||
docker compose up -d nextcloud nextcloud-db webdav smb minio
|
||||
docker compose up -d seafile-db seafile-memcached seafile
|
||||
```
|
||||
|
||||
@@ -29,14 +30,17 @@ Endpoints:
|
||||
`http://127.0.0.1:9082/seafdav/`
|
||||
- WebDAV: `http://127.0.0.1:9083`
|
||||
- SMB: `smb://127.0.0.1:1445/files`
|
||||
- MinIO/S3 API: `http://127.0.0.1:9000`, console
|
||||
`http://127.0.0.1:9001`
|
||||
|
||||
The local fixture data under `data/` is ignored by git.
|
||||
|
||||
## GovOPlaN Profile Config
|
||||
|
||||
Connector profiles are read from `GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON` or
|
||||
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. Credentials should be referenced by
|
||||
secret refs or environment variables; profile API responses only expose the
|
||||
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. This deployment-owned configuration
|
||||
may reference environment variables only when their exact names are listed in
|
||||
`GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST`; profile API responses expose only the
|
||||
credential source and configured state.
|
||||
|
||||
Example local profile file:
|
||||
@@ -92,6 +96,25 @@ Example local profile file:
|
||||
"password_env": "SMB_PASSWORD",
|
||||
"capabilities": ["browse", "import"],
|
||||
"policy": { "allow": { "providers": ["smb"] } }
|
||||
},
|
||||
{
|
||||
"id": "dev-s3",
|
||||
"label": "Dev MinIO",
|
||||
"provider": "s3",
|
||||
"endpoint_url": "http://127.0.0.1:9000",
|
||||
"base_path": "",
|
||||
"scope_type": "system",
|
||||
"credential_mode": "basic",
|
||||
"username": "govoplan",
|
||||
"password_env": "MINIO_ROOT_PASSWORD",
|
||||
"capabilities": ["browse", "import"],
|
||||
"metadata": {
|
||||
"bucket": "govoplan",
|
||||
"region": "us-east-1",
|
||||
"path_style": true,
|
||||
"verify_tls": false
|
||||
},
|
||||
"policy": { "allow": { "providers": ["s3"] } }
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -101,6 +124,8 @@ Start GovOPlaN with:
|
||||
|
||||
```bash
|
||||
export GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE=/mnt/DATA/git/govoplan-files/dev/connectors/profiles.local.json
|
||||
export GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST=SEAFILE_ADMIN_PASSWORD,NEXTCLOUD_ADMIN_PASSWORD,WEBDAV_PASSWORD,SMB_PASSWORD,MINIO_ROOT_PASSWORD
|
||||
export GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS=true
|
||||
```
|
||||
|
||||
## Smoke Test
|
||||
@@ -113,10 +138,13 @@ cd /mnt/DATA/git/govoplan-files/dev/connectors
|
||||
/mnt/DATA/git/govoplan-core/.venv/bin/python smoke.py
|
||||
```
|
||||
|
||||
The script reads `.env` from this directory when present, seeds tiny WebDAV,
|
||||
Nextcloud, and SMB fixtures, then browses and imports them through the connector
|
||||
helper layer. Pass `--require-smb` after recreating the SMB container to fail on
|
||||
SMB access errors instead of reporting them as an optional skip.
|
||||
The script reads `.env` from this directory when present and seeds tiny WebDAV,
|
||||
Nextcloud, SMB, and MinIO fixtures. WebDAV and Nextcloud are browsed and imported
|
||||
through the pinned connector transport. SMB and S3 product access deliberately
|
||||
fails closed until their SDK transports support peer pinning, so their fixtures
|
||||
are retained for transport development and reported as expected optional skips.
|
||||
The `--require-smb` and `--require-s3` switches are useful only while developing
|
||||
that transport support and currently make the smoke check fail by design.
|
||||
|
||||
SMB smoke checks need the optional Python dependency in the environment running
|
||||
the script:
|
||||
@@ -125,6 +153,20 @@ the script:
|
||||
/mnt/DATA/git/govoplan-core/.venv/bin/python -m pip install -e /mnt/DATA/git/govoplan-files[smb]
|
||||
```
|
||||
|
||||
S3 smoke checks need the optional boto3 dependency in the environment running
|
||||
the script:
|
||||
|
||||
```bash
|
||||
/mnt/DATA/git/govoplan-core/.venv/bin/python -m pip install -e /mnt/DATA/git/govoplan-files[s3]
|
||||
```
|
||||
|
||||
The SMB service is built from `dev/connectors/smb/` so the development share is
|
||||
deterministic: one `files` share backed by `data/smb`, with the credentials from
|
||||
`.env`.
|
||||
|
||||
The SMB image runs as the non-root `govoplan` user with UID/GID `1000` and
|
||||
listens on unprivileged container port `1445`. The default host endpoint remains
|
||||
`smb://127.0.0.1:1445/files`. `SMB_USER` must stay `govoplan` unless the image is
|
||||
rebuilt with a matching user; `SMB_PORT` must be `1024` or higher. `SMB_PASSWORD`
|
||||
is applied when the image is built, so rebuild the `smb` service after changing
|
||||
it in `.env`.
|
||||
|
||||
@@ -86,21 +86,35 @@ services:
|
||||
smb:
|
||||
build:
|
||||
context: ./smb
|
||||
args:
|
||||
SMB_PASSWORD: ${SMB_PASSWORD:-govoplan-smb}
|
||||
image: govoplan-files-samba-dev:latest
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
SMB_SHARE_NAME: ${SMB_SHARE_NAME:-files}
|
||||
SMB_USER: ${SMB_USER:-govoplan}
|
||||
SMB_PASSWORD: ${SMB_PASSWORD:-govoplan-smb}
|
||||
SMB_UID: ${SMB_UID:-1000}
|
||||
SMB_GID: ${SMB_GID:-1000}
|
||||
SMB_PORT: ${SMB_PORT:-1445}
|
||||
ports:
|
||||
- "127.0.0.1:${SMB_HOST_PORT:-1445}:445"
|
||||
- "127.0.0.1:${SMB_HOST_PORT:-1445}:${SMB_PORT:-1445}"
|
||||
volumes:
|
||||
- ./data/smb:/storage
|
||||
|
||||
minio:
|
||||
image: minio/minio:latest
|
||||
restart: unless-stopped
|
||||
command: server /data --console-address ":9001"
|
||||
environment:
|
||||
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-govoplan}
|
||||
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-govoplan-minio}
|
||||
ports:
|
||||
- "127.0.0.1:${MINIO_API_HOST_PORT:-9000}:9000"
|
||||
- "127.0.0.1:${MINIO_CONSOLE_HOST_PORT:-9001}:9001"
|
||||
volumes:
|
||||
- minio:/data
|
||||
|
||||
volumes:
|
||||
nextcloud-db:
|
||||
nextcloud:
|
||||
seafile-db:
|
||||
seafile-data:
|
||||
minio:
|
||||
|
||||
@@ -1,10 +1,29 @@
|
||||
FROM alpine:3.20
|
||||
|
||||
RUN apk add --no-cache samba-server samba-common-tools
|
||||
ARG SMB_PASSWORD=govoplan-smb
|
||||
|
||||
RUN apk add --no-cache samba-server samba-common-tools \
|
||||
&& addgroup -S -g 1000 govoplan \
|
||||
&& adduser -S -D -H -h /nonexistent -s /sbin/nologin -u 1000 -G govoplan govoplan \
|
||||
&& mkdir -p /storage /var/lib/samba/private /var/cache/samba /run/samba /etc/samba /var/log/samba/cores \
|
||||
&& printf '%s\n' \
|
||||
'[global]' \
|
||||
' passdb backend = tdbsam' \
|
||||
' private dir = /var/lib/samba/private' \
|
||||
' lock directory = /run/samba' \
|
||||
' state directory = /var/lib/samba' \
|
||||
' cache directory = /var/cache/samba' \
|
||||
' pid directory = /run/samba' \
|
||||
> /etc/samba/smb.conf \
|
||||
&& printf '%s\n%s\n' "$SMB_PASSWORD" "$SMB_PASSWORD" | smbpasswd -s -a govoplan \
|
||||
&& smbpasswd -e govoplan \
|
||||
&& chown -R govoplan:govoplan /storage /var/lib/samba /var/cache/samba /run/samba /etc/samba /var/log/samba
|
||||
|
||||
COPY entrypoint.sh /usr/local/bin/govoplan-samba-entrypoint
|
||||
RUN chmod +x /usr/local/bin/govoplan-samba-entrypoint
|
||||
|
||||
EXPOSE 445
|
||||
EXPOSE 1445
|
||||
|
||||
USER govoplan:govoplan
|
||||
|
||||
ENTRYPOINT ["/usr/local/bin/govoplan-samba-entrypoint"]
|
||||
|
||||
@@ -3,21 +3,42 @@ set -eu
|
||||
|
||||
share_name="${SMB_SHARE_NAME:-files}"
|
||||
user_name="${SMB_USER:-govoplan}"
|
||||
password="${SMB_PASSWORD:-govoplan-smb}"
|
||||
uid="${SMB_UID:-1000}"
|
||||
gid="${SMB_GID:-1000}"
|
||||
smb_port="${SMB_PORT:-1445}"
|
||||
runtime_user="$(id -un)"
|
||||
runtime_group="$(id -gn)"
|
||||
|
||||
mkdir -p /storage /var/lib/samba/private /run/samba
|
||||
case "$share_name" in
|
||||
"" | *[!A-Za-z0-9_.-]*)
|
||||
printf 'SMB_SHARE_NAME must contain only letters, numbers, dot, dash, or underscore.\n' >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
if ! getent group "$user_name" >/dev/null 2>&1; then
|
||||
addgroup -g "$gid" "$user_name"
|
||||
case "$user_name" in
|
||||
"" | *[!A-Za-z0-9_.-]*)
|
||||
printf 'SMB_USER must contain only letters, numbers, dot, dash, or underscore.\n' >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
case "$smb_port" in
|
||||
"" | *[!0-9]*)
|
||||
printf 'SMB_PORT must be numeric.\n' >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
if [ "$smb_port" -lt 1024 ]; then
|
||||
printf 'SMB_PORT must be >= 1024 because the dev Samba container runs as a non-root user.\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! id "$user_name" >/dev/null 2>&1; then
|
||||
adduser -D -H -s /sbin/nologin -u "$uid" -G "$user_name" "$user_name"
|
||||
if [ "$user_name" != "$runtime_user" ]; then
|
||||
printf 'SMB_USER=%s is not supported by the non-root dev image; use %s or rebuild the image with a matching user.\n' "$user_name" "$runtime_user" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
chown -R "$user_name:$user_name" /storage
|
||||
mkdir -p /storage /var/lib/samba/private /var/cache/samba /run/samba /etc/samba
|
||||
|
||||
cat > /etc/samba/smb.conf <<EOF
|
||||
[global]
|
||||
@@ -25,12 +46,20 @@ cat > /etc/samba/smb.conf <<EOF
|
||||
workgroup = WORKGROUP
|
||||
security = user
|
||||
map to guest = Never
|
||||
guest account = $runtime_user
|
||||
server min protocol = SMB2
|
||||
server signing = mandatory
|
||||
smb ports = $smb_port
|
||||
load printers = no
|
||||
printing = bsd
|
||||
disable spoolss = yes
|
||||
log level = 1
|
||||
passdb backend = tdbsam
|
||||
private dir = /var/lib/samba/private
|
||||
lock directory = /run/samba
|
||||
state directory = /var/lib/samba
|
||||
cache directory = /var/cache/samba
|
||||
pid directory = /run/samba
|
||||
|
||||
[$share_name]
|
||||
path = /storage
|
||||
@@ -38,13 +67,16 @@ cat > /etc/samba/smb.conf <<EOF
|
||||
read only = no
|
||||
guest ok = no
|
||||
valid users = $user_name
|
||||
force user = $user_name
|
||||
force group = $user_name
|
||||
force user = $runtime_user
|
||||
force group = $runtime_group
|
||||
create mask = 0664
|
||||
directory mask = 0775
|
||||
EOF
|
||||
|
||||
printf '%s\n%s\n' "$password" "$password" | smbpasswd -s -a "$user_name" >/dev/null
|
||||
smbpasswd -e "$user_name" >/dev/null
|
||||
if ! pdbedit -L -u "$user_name" >/dev/null 2>&1; then
|
||||
printf 'Samba user %s is missing from passdb. Rebuild the smb image so the non-root passdb is seeded.\n' "$user_name" >&2
|
||||
printf 'Run: docker compose build --no-cache smb && docker compose up -d smb\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exec smbd --foreground --no-process-group --debug-stdout
|
||||
exec smbd --foreground --no-process-group --debug-stdout -p "$smb_port"
|
||||
|
||||
@@ -18,16 +18,17 @@ ROOT = Path(__file__).resolve().parent
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Smoke-test GovOPlaN connector helpers against the local dev compose stack.")
|
||||
parser.add_argument("--require-smb", action="store_true", help="Fail if the SMB connector cannot browse/import.")
|
||||
parser.add_argument("--require-s3", action="store_true", help="Fail if the S3 connector cannot browse/import.")
|
||||
parser.add_argument("--debug-smb", action="store_true", help="Print direct smbclient probes before the connector smoke check.")
|
||||
args = parser.parse_args()
|
||||
|
||||
_load_dotenv()
|
||||
_default_env()
|
||||
preflight_failures = _preflight_services(require_smb=args.require_smb)
|
||||
preflight_failures = _preflight_services(require_smb=args.require_smb, require_s3=args.require_s3)
|
||||
if preflight_failures:
|
||||
for failure in preflight_failures:
|
||||
print(f"FAIL {failure}")
|
||||
print("Start or recreate the connector stack from this directory with: docker compose up -d nextcloud nextcloud-db webdav smb")
|
||||
print("Start or recreate the connector stack from this directory with: docker compose up -d nextcloud nextcloud-db webdav smb minio")
|
||||
return 1
|
||||
_seed_webdav_fixture()
|
||||
_seed_smb_fixture()
|
||||
@@ -36,6 +37,13 @@ def main() -> int:
|
||||
except httpx.HTTPError as exc:
|
||||
print(f"FAIL dev-nextcloud seed failed: {exc}")
|
||||
return 1
|
||||
try:
|
||||
_seed_s3_fixture()
|
||||
except Exception as exc:
|
||||
if args.require_s3:
|
||||
print(f"FAIL dev-s3 seed failed: {exc}")
|
||||
return 1
|
||||
print(f"SKIP dev-s3 seed failed: {exc}")
|
||||
|
||||
profiles = {profile.id: profile for profile in connector_profiles_from_payload({"profiles": _profile_payloads()})}
|
||||
if args.debug_smb:
|
||||
@@ -59,6 +67,15 @@ def main() -> int:
|
||||
else:
|
||||
print(f"SKIP {message}")
|
||||
|
||||
try:
|
||||
_exercise_profile(profiles["dev-s3"], folder="GovOPlaN", file_path="GovOPlaN/s3-live.txt", expected="s3 live fixture")
|
||||
except Exception as exc:
|
||||
message = f"dev-s3: {exc}"
|
||||
if args.require_s3:
|
||||
failures.append(message)
|
||||
else:
|
||||
print(f"SKIP {message}")
|
||||
|
||||
if failures:
|
||||
for failure in failures:
|
||||
print(f"FAIL {failure}")
|
||||
@@ -76,6 +93,9 @@ def _default_env() -> None:
|
||||
"SMB_SHARE_NAME": "files",
|
||||
"SMB_USER": "govoplan",
|
||||
"SMB_PASSWORD": "govoplan-smb",
|
||||
"MINIO_ROOT_USER": "govoplan",
|
||||
"MINIO_ROOT_PASSWORD": "govoplan-minio",
|
||||
"MINIO_BUCKET": "govoplan",
|
||||
}
|
||||
for key, value in defaults.items():
|
||||
os.environ.setdefault(key, value)
|
||||
@@ -96,7 +116,7 @@ def _load_dotenv() -> None:
|
||||
os.environ[key] = value.strip().strip("\"'")
|
||||
|
||||
|
||||
def _preflight_services(*, require_smb: bool) -> list[str]:
|
||||
def _preflight_services(*, require_smb: bool, require_s3: bool) -> list[str]:
|
||||
failures: list[str] = []
|
||||
nextcloud_url = f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/status.php"
|
||||
try:
|
||||
@@ -121,6 +141,14 @@ def _preflight_services(*, require_smb: bool) -> list[str]:
|
||||
pass
|
||||
except OSError as exc:
|
||||
failures.append(f"dev-smb is not reachable at 127.0.0.1:{smb_port}: {exc}")
|
||||
if require_s3:
|
||||
minio_url = f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}/minio/health/live"
|
||||
try:
|
||||
response = httpx.get(minio_url, timeout=3.0)
|
||||
if response.status_code != 200:
|
||||
failures.append(f"dev-s3 expected HTTP 200 at {minio_url}, got HTTP {response.status_code}")
|
||||
except httpx.HTTPError as exc:
|
||||
failures.append(f"dev-s3 is not reachable at {minio_url}: {exc}")
|
||||
return failures
|
||||
|
||||
|
||||
@@ -150,6 +178,20 @@ def _profile_payloads() -> list[dict[str, object]]:
|
||||
"username": os.getenv("SMB_USER", "govoplan"),
|
||||
"password_env": "SMB_PASSWORD",
|
||||
},
|
||||
{
|
||||
"id": "dev-s3",
|
||||
"provider": "s3",
|
||||
"endpoint_url": f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}",
|
||||
"credential_mode": "basic",
|
||||
"username": os.getenv("MINIO_ROOT_USER", "govoplan"),
|
||||
"password_env": "MINIO_ROOT_PASSWORD",
|
||||
"metadata": {
|
||||
"bucket": os.getenv("MINIO_BUCKET", "govoplan"),
|
||||
"region": "us-east-1",
|
||||
"path_style": True,
|
||||
"verify_tls": False,
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
@@ -211,5 +253,30 @@ def _seed_nextcloud_fixture() -> None:
|
||||
raise RuntimeError(f"Nextcloud PUT failed with HTTP {response.status_code}: {response.text[:200]}")
|
||||
|
||||
|
||||
def _seed_s3_fixture() -> None:
|
||||
try:
|
||||
import boto3
|
||||
from botocore.config import Config
|
||||
from botocore.exceptions import ClientError
|
||||
except ImportError as exc:
|
||||
raise RuntimeError("boto3 is not installed; install govoplan-files[s3]") from exc
|
||||
bucket = os.getenv("MINIO_BUCKET", "govoplan")
|
||||
client = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=f"http://127.0.0.1:{os.getenv('MINIO_API_HOST_PORT', '9000')}",
|
||||
aws_access_key_id=os.getenv("MINIO_ROOT_USER", "govoplan"),
|
||||
aws_secret_access_key=os.getenv("MINIO_ROOT_PASSWORD", "govoplan-minio"),
|
||||
region_name="us-east-1",
|
||||
config=Config(s3={"addressing_style": "path"}),
|
||||
)
|
||||
try:
|
||||
client.create_bucket(Bucket=bucket)
|
||||
except ClientError as exc:
|
||||
code = exc.response.get("Error", {}).get("Code")
|
||||
if code not in {"BucketAlreadyOwnedByYou", "BucketAlreadyExists"}:
|
||||
raise
|
||||
client.put_object(Bucket=bucket, Key="GovOPlaN/s3-live.txt", Body=b"s3 live fixture\n", ContentType="text/plain")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
||||
@@ -49,13 +49,22 @@ any remote write behavior.
|
||||
Every provider must:
|
||||
|
||||
- enforce GovOPlaN profile visibility and connector policy before browse/import
|
||||
- keep credentials as environment variables or secret references, never API
|
||||
response values
|
||||
- keep credentials as encrypted values or scoped secret references, never API
|
||||
response values; process-environment references are allowed only in
|
||||
deployment-owned profiles with an exact deployment allowlist
|
||||
- import external files into managed storage before they are used in campaigns
|
||||
or workflows
|
||||
- preserve source provenance and revision metadata
|
||||
- emit connector audit events for imported or accessed files
|
||||
- treat remote ACLs as upstream checks, not as a replacement for GovOPlaN policy
|
||||
- use a transport that pins every connection to a policy-validated DNS/IP answer
|
||||
and revalidates redirects; SDK transports without that guarantee fail closed
|
||||
|
||||
Live SMB access is disabled in all modes until `smbprotocol` initial connections
|
||||
and DFS referral targets can be pinned and policy-validated. An explicit IP is
|
||||
not sufficient because the server may still issue a referral to another peer.
|
||||
Live S3 access is likewise disabled until `boto3`/botocore can be bound to the
|
||||
pinned transport, including SDK-managed redirects and endpoint discovery.
|
||||
|
||||
## Non-Goals For Files
|
||||
|
||||
|
||||
@@ -34,8 +34,9 @@ Credential profile:
|
||||
|
||||
- provider: a specific provider or any provider
|
||||
- scope: `system` or `tenant` for administered credentials
|
||||
- credential mode: anonymous, environment reference, secret reference, or
|
||||
encrypted stored password/token
|
||||
- credential mode: anonymous, scoped secret reference, or encrypted stored
|
||||
password/token; environment references are reserved for deployment-owned JSON
|
||||
profiles and exact deployment allowlisting
|
||||
- username and redacted secret configuration
|
||||
- credential-local policy for allow/deny rules
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@govoplan/files-webui",
|
||||
"version": "0.1.8",
|
||||
"version": "0.1.9",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "webui/src/index.ts",
|
||||
@@ -19,7 +19,7 @@
|
||||
"LICENSE"
|
||||
],
|
||||
"peerDependencies": {
|
||||
"@govoplan/core-webui": "^0.1.8",
|
||||
"@govoplan/core-webui": "^0.1.9",
|
||||
"lucide-react": "^1.23.0",
|
||||
"react": "^19.0.0",
|
||||
"react-dom": "^19.0.0",
|
||||
|
||||
@@ -4,19 +4,22 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "govoplan-files"
|
||||
version = "0.1.8"
|
||||
version = "0.1.9"
|
||||
description = "GovOPlaN files module with backend and WebUI integration."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
license = { file = "LICENSE" }
|
||||
authors = [{ name = "GovOPlaN" }]
|
||||
dependencies = [
|
||||
"govoplan-core>=0.1.8",
|
||||
"govoplan-core>=0.1.9",
|
||||
"defusedxml>=0.7,<1",
|
||||
"python-multipart>=0.0.31,<1",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
s3 = [
|
||||
"boto3>=1.34,<2",
|
||||
]
|
||||
smb = [
|
||||
"smbprotocol>=1.13",
|
||||
]
|
||||
|
||||
@@ -17,6 +17,7 @@ from govoplan_files.backend.storage.campaign_attachments import (
|
||||
managed_match_payloads,
|
||||
prepared_campaign_snapshot,
|
||||
public_attachment_summary_payload,
|
||||
share_assets_with_campaign,
|
||||
)
|
||||
from govoplan_files.backend.storage.campaign_usage import record_campaign_attachment_uses_for_jobs
|
||||
from govoplan_files.backend.storage.files import current_version_and_blob
|
||||
@@ -44,6 +45,7 @@ class FilesCampaignCapability:
|
||||
annotate_built_messages_with_managed_files = staticmethod(annotate_built_messages_with_managed_files)
|
||||
record_campaign_attachment_uses_for_jobs = staticmethod(record_campaign_attachment_uses_for_jobs)
|
||||
current_version_and_blob = staticmethod(current_version_and_blob)
|
||||
share_assets_with_campaign = staticmethod(share_assets_with_campaign)
|
||||
|
||||
@staticmethod
|
||||
def mark_job_attachment_uses_sent(session: Any, job: Any) -> None:
|
||||
|
||||
@@ -1,12 +1,18 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import event, inspect
|
||||
from sqlalchemy import event
|
||||
from sqlalchemy.orm import Session as OrmSession
|
||||
|
||||
from govoplan_core.core.change_sequence import record_change
|
||||
from govoplan_core.core.sqlalchemy_change_tracking import (
|
||||
ensure_object_id,
|
||||
has_attr_changes,
|
||||
object_state,
|
||||
operation_for_soft_deletable,
|
||||
previous_value,
|
||||
)
|
||||
from govoplan_files.backend.db.models import FileAsset, FileFolder, FileShare, new_uuid
|
||||
|
||||
FILES_MODULE_ID = "files"
|
||||
@@ -39,7 +45,7 @@ def _record_files_changes(session: OrmSession, _flush_context: object, _instance
|
||||
|
||||
|
||||
def _record_asset_change(session: OrmSession, asset: FileAsset) -> None:
|
||||
operation = _operation_for_soft_deletable(
|
||||
operation = operation_for_soft_deletable(
|
||||
asset,
|
||||
changed_attrs=(
|
||||
"owner_type",
|
||||
@@ -70,7 +76,7 @@ def _record_asset_change(session: OrmSession, asset: FileAsset) -> None:
|
||||
"owner_type": asset.owner_type,
|
||||
"owner_id": _owner_id(asset.owner_type, asset.owner_user_id, asset.owner_group_id),
|
||||
"path": asset.display_path,
|
||||
"previous_path": _previous_value(asset, "display_path"),
|
||||
"previous_path": previous_value(asset, "display_path"),
|
||||
"filename": asset.filename,
|
||||
"deleted_at": _isoformat(asset.deleted_at),
|
||||
},
|
||||
@@ -78,7 +84,7 @@ def _record_asset_change(session: OrmSession, asset: FileAsset) -> None:
|
||||
|
||||
|
||||
def _record_folder_change(session: OrmSession, folder: FileFolder) -> None:
|
||||
operation = _operation_for_soft_deletable(
|
||||
operation = operation_for_soft_deletable(
|
||||
folder,
|
||||
changed_attrs=("owner_type", "owner_user_id", "owner_group_id", "path", "deleted_at", "metadata_"),
|
||||
)
|
||||
@@ -99,17 +105,17 @@ def _record_folder_change(session: OrmSession, folder: FileFolder) -> None:
|
||||
"owner_type": folder.owner_type,
|
||||
"owner_id": _owner_id(folder.owner_type, folder.owner_user_id, folder.owner_group_id),
|
||||
"path": folder.path,
|
||||
"previous_path": _previous_value(folder, "path"),
|
||||
"previous_path": previous_value(folder, "path"),
|
||||
"deleted_at": _isoformat(folder.deleted_at),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _record_share_visibility_change(session: OrmSession, share: FileShare) -> None:
|
||||
state = inspect(share)
|
||||
state = object_state(share)
|
||||
if not share.file_asset_id:
|
||||
return
|
||||
if not state.pending and not _has_attr_changes(
|
||||
if not state.pending and not has_attr_changes(
|
||||
state,
|
||||
("file_asset_id", "target_type", "target_id", "permission", "revoked_at"),
|
||||
):
|
||||
@@ -135,44 +141,8 @@ def _record_share_visibility_change(session: OrmSession, share: FileShare) -> No
|
||||
)
|
||||
|
||||
|
||||
def _operation_for_soft_deletable(obj: object, *, changed_attrs: tuple[str, ...]) -> str | None:
|
||||
state = inspect(obj)
|
||||
if state.pending:
|
||||
return "created"
|
||||
if not _has_attr_changes(state, changed_attrs):
|
||||
return None
|
||||
if "deleted_at" in state.attrs:
|
||||
history = state.attrs.deleted_at.history
|
||||
if history.has_changes():
|
||||
if any(value is not None for value in history.added):
|
||||
return "deleted"
|
||||
if any(value is not None for value in history.deleted):
|
||||
return "created"
|
||||
return "updated"
|
||||
|
||||
|
||||
def _has_attr_changes(state: Any, attrs: tuple[str, ...]) -> bool:
|
||||
return any(name in state.attrs and state.attrs[name].history.has_changes() for name in attrs)
|
||||
|
||||
|
||||
def _previous_value(obj: object, attr_name: str) -> str | None:
|
||||
state = inspect(obj)
|
||||
if attr_name not in state.attrs:
|
||||
return None
|
||||
history = state.attrs[attr_name].history
|
||||
if not history.has_changes() or not history.deleted:
|
||||
return None
|
||||
value = history.deleted[0]
|
||||
return str(value) if value is not None else None
|
||||
|
||||
|
||||
def _ensure_id(obj: object) -> str:
|
||||
resource_id = getattr(obj, "id", None)
|
||||
if resource_id:
|
||||
return str(resource_id)
|
||||
resource_id = new_uuid()
|
||||
setattr(obj, "id", resource_id)
|
||||
return resource_id
|
||||
return ensure_object_id(obj, new_uuid)
|
||||
|
||||
|
||||
def _owner_id(owner_type: str, owner_user_id: str | None, owner_group_id: str | None) -> str | None:
|
||||
|
||||
@@ -1 +1,25 @@
|
||||
from govoplan_files.backend.db.models import *
|
||||
from govoplan_files.backend.db.models import (
|
||||
CampaignAttachmentUse,
|
||||
FileAsset,
|
||||
FileBlob,
|
||||
FileConnectorCredential,
|
||||
FileConnectorPolicy,
|
||||
FileConnectorProfile,
|
||||
FileConnectorSpace,
|
||||
FileFolder,
|
||||
FileShare,
|
||||
FileVersion,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"CampaignAttachmentUse",
|
||||
"FileAsset",
|
||||
"FileBlob",
|
||||
"FileConnectorCredential",
|
||||
"FileConnectorPolicy",
|
||||
"FileConnectorProfile",
|
||||
"FileConnectorSpace",
|
||||
"FileFolder",
|
||||
"FileShare",
|
||||
"FileVersion",
|
||||
]
|
||||
|
||||
@@ -1,11 +1,17 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import inspect
|
||||
|
||||
from govoplan_core.core.access import CAPABILITY_AUTH_PERMISSION_EVALUATOR, CAPABILITY_AUTH_PRINCIPAL_RESOLVER
|
||||
from govoplan_core.core.files import CAPABILITY_FILES_ACCESS
|
||||
from govoplan_core.core.module_guards import drop_table_retirement_provider, persistent_table_uninstall_guard
|
||||
from govoplan_core.core.modules import (
|
||||
DocumentationCondition,
|
||||
DocumentationLink,
|
||||
DocumentationTopic,
|
||||
FrontendModule,
|
||||
MigrationSpec,
|
||||
ModuleContext,
|
||||
@@ -23,6 +29,55 @@ from govoplan_files.backend.db import models as file_models # noqa: F401 - popu
|
||||
register_files_change_tracking()
|
||||
|
||||
|
||||
_files_table_retirement_provider = drop_table_retirement_provider(
|
||||
file_models.FileBlob,
|
||||
file_models.FileFolder,
|
||||
file_models.FileAsset,
|
||||
file_models.FileVersion,
|
||||
file_models.FileShare,
|
||||
file_models.FileConnectorCredential,
|
||||
file_models.FileConnectorPolicy,
|
||||
file_models.FileConnectorProfile,
|
||||
file_models.FileConnectorSpace,
|
||||
file_models.CampaignAttachmentUse,
|
||||
label="Files",
|
||||
)
|
||||
|
||||
|
||||
def _files_retirement_provider(session: object | None, module_id: str):
|
||||
plan = _files_table_retirement_provider(session, module_id)
|
||||
base_executor = plan.destroy_data_executor
|
||||
if base_executor is None:
|
||||
return plan
|
||||
|
||||
def executor(execute_session: object, execute_module_id: str) -> None:
|
||||
if not hasattr(execute_session, "get_bind") or not hasattr(execute_session, "query"):
|
||||
raise RuntimeError("No database session is available for Files credential retirement.")
|
||||
live_inspector = inspect(execute_session.get_bind())
|
||||
if any(
|
||||
live_inspector.has_table(table_name)
|
||||
for table_name in (
|
||||
file_models.FileConnectorCredential.__tablename__,
|
||||
file_models.FileConnectorProfile.__tablename__,
|
||||
)
|
||||
):
|
||||
from govoplan_files.backend.storage.connector_credential_deletion import (
|
||||
delete_connector_credentials_for_retirement,
|
||||
)
|
||||
|
||||
delete_connector_credentials_for_retirement(execute_session)
|
||||
base_executor(execute_session, execute_module_id)
|
||||
|
||||
return replace(
|
||||
plan,
|
||||
destroy_data_warnings=(
|
||||
*plan.destroy_data_warnings,
|
||||
"Files-owned encrypted connector credentials are scrubbed and audited immediately before tables are dropped; legacy non-owned external references are detached without claiming provider-side deletion.",
|
||||
),
|
||||
destroy_data_executor=executor,
|
||||
)
|
||||
|
||||
|
||||
def _permission(scope: str, label: str, description: str) -> PermissionDefinition:
|
||||
module_id, resource, action = scope.split(":", 2)
|
||||
return PermissionDefinition(
|
||||
@@ -112,7 +167,7 @@ def _files_router(context: ModuleContext):
|
||||
manifest = ModuleManifest(
|
||||
id="files",
|
||||
name="Files",
|
||||
version="0.1.8",
|
||||
version="0.1.9",
|
||||
required_capabilities=(CAPABILITY_AUTH_PRINCIPAL_RESOLVER, CAPABILITY_AUTH_PERMISSION_EVALUATOR),
|
||||
optional_dependencies=("campaigns",),
|
||||
provides_interfaces=(
|
||||
@@ -138,24 +193,83 @@ manifest = ModuleManifest(
|
||||
package_name="@govoplan/files-webui",
|
||||
nav_items=(NavItem(path="/files", label="Files", icon="folder", required_any=("files:file:read",), order=40),),
|
||||
),
|
||||
documentation=(
|
||||
DocumentationTopic(
|
||||
id="files.governed-connectors-and-provenance",
|
||||
title="Governed file connectors and source provenance",
|
||||
summary="Connector profiles control who may browse an external source, which destinations are permitted, and how imported files retain verifiable source context.",
|
||||
body=(
|
||||
"Users select only connector profiles visible in their current scope and import remote content into managed Files storage before another module uses it. "
|
||||
"Administrators define profiles, separate credential references, and ordered system/tenant/owner policies; deny rules win and profile responses never expose secrets. "
|
||||
"Deleting a database-managed profile or credential immediately scrubs Files-owned encrypted material and records a non-secret audit event in the same transaction; legacy non-owned external references are detached without provider calls. "
|
||||
"Operators control private-network access deployment-wide and must keep every remote connection pinned to a policy-validated DNS/IP answer. "
|
||||
"The built-in HTTP transport pins each connection and refuses redirects. Live S3 and SMB SDK access fails closed until S3 redirects and SMB DFS referrals can be revalidated and pinned. "
|
||||
"Successful imports store the connector id, provider, remote path and identity, source revision, and selected metadata as provenance on the managed file and its audit events."
|
||||
),
|
||||
layer="configured",
|
||||
documentation_types=("admin", "user"),
|
||||
audience=("file_user", "file_admin", "tenant_admin", "operator"),
|
||||
order=42,
|
||||
conditions=(
|
||||
DocumentationCondition(
|
||||
required_modules=("files",),
|
||||
any_scopes=(
|
||||
"files:file:read",
|
||||
"files:file:upload",
|
||||
"files:file:admin",
|
||||
"admin:settings:read",
|
||||
"system:settings:read",
|
||||
),
|
||||
),
|
||||
),
|
||||
links=(
|
||||
DocumentationLink(label="Files", href="/files", kind="runtime"),
|
||||
DocumentationLink(label="Tenant connector administration", href="/admin?section=tenant-file-connectors", kind="runtime"),
|
||||
DocumentationLink(label="Connector profiles API", href="/api/v1/files/connectors/profiles", kind="api"),
|
||||
DocumentationLink(label="Connector providers API", href="/api/v1/files/connectors/providers", kind="api"),
|
||||
DocumentationLink(label="Tenant connector policy API", href="/api/v1/files/connectors/policies/tenant", kind="api"),
|
||||
),
|
||||
related_modules=("campaigns",),
|
||||
unlocks=("Campaigns can consume managed attachments while Files preserves frozen source provenance.",),
|
||||
configuration_keys=(
|
||||
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS",
|
||||
"GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON",
|
||||
"GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE",
|
||||
"GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST",
|
||||
),
|
||||
metadata={
|
||||
"kind": "reference",
|
||||
"perspectives": {
|
||||
"user": "Choose a visible connection, browse it read-only, and import a selected file into managed storage before using it elsewhere.",
|
||||
"admin": "Govern profile visibility, separate credential references, provider allow/deny policy, and permitted connector operations.",
|
||||
"operator": "Control private-network access deployment-wide, provide credential and CA allowlists, and investigate fail-closed transport errors without weakening peer validation.",
|
||||
},
|
||||
"security_invariants": [
|
||||
"Every network peer must be policy-validated and pinned at connection time.",
|
||||
"Redirects and protocol referrals must be rejected or independently revalidated and pinned.",
|
||||
"SDK transports that cannot provide those guarantees fail before client construction.",
|
||||
"New API-managed external secret references fail closed until Files can prove ownership and provider-side deletion.",
|
||||
"Deletion and destructive retirement scrub Files-owned encrypted connector material before completion and emit non-secret audit evidence.",
|
||||
"Legacy non-owned external references are detached and audited, never sent to an arbitrary provider delete operation.",
|
||||
],
|
||||
"provenance_fields": [
|
||||
"connector_id",
|
||||
"provider",
|
||||
"external_id",
|
||||
"external_path",
|
||||
"revision",
|
||||
"metadata",
|
||||
],
|
||||
"related_topic_ids": ["mail.profiles-and-policy"],
|
||||
},
|
||||
),
|
||||
),
|
||||
migration_spec=MigrationSpec(
|
||||
module_id="files",
|
||||
metadata=Base.metadata,
|
||||
script_location=str(Path(__file__).with_name("migrations") / "versions"),
|
||||
retirement_supported=True,
|
||||
retirement_provider=drop_table_retirement_provider(
|
||||
file_models.FileBlob,
|
||||
file_models.FileFolder,
|
||||
file_models.FileAsset,
|
||||
file_models.FileVersion,
|
||||
file_models.FileShare,
|
||||
file_models.FileConnectorCredential,
|
||||
file_models.FileConnectorPolicy,
|
||||
file_models.FileConnectorProfile,
|
||||
file_models.FileConnectorSpace,
|
||||
file_models.CampaignAttachmentUse,
|
||||
label="Files",
|
||||
),
|
||||
retirement_provider=_files_retirement_provider,
|
||||
retirement_notes="Destructive retirement drops files-owned database tables after the installer captures a database snapshot.",
|
||||
),
|
||||
uninstall_guard_providers=(
|
||||
|
||||
@@ -7,7 +7,7 @@ from dataclasses import replace
|
||||
from datetime import datetime
|
||||
from io import BytesIO
|
||||
from typing import Any, Literal
|
||||
from urllib.parse import quote, urljoin
|
||||
from urllib.parse import quote, urljoin, urlsplit, urlunsplit
|
||||
from fastapi import APIRouter, Depends, File as FastAPIFile, Form, HTTPException, Query, UploadFile, status
|
||||
from fastapi.responses import FileResponse, StreamingResponse
|
||||
from starlette.background import BackgroundTask
|
||||
@@ -105,13 +105,17 @@ from govoplan_files.backend.storage.access import ensure_group_access, group_ref
|
||||
from govoplan_files.backend.storage.archives import create_zip_file, extract_zip_upload
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_credential_store import (
|
||||
ConnectorCredential,
|
||||
connector_credential_from_row,
|
||||
create_connector_credential_row,
|
||||
deactivate_connector_credential_row,
|
||||
get_connector_credential_row,
|
||||
list_database_connector_credentials,
|
||||
update_connector_credential_row,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_credential_deletion import (
|
||||
delete_connector_credential_row,
|
||||
delete_connector_profile_row,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
ConnectorBrowseError,
|
||||
ConnectorBrowseUnsupported,
|
||||
@@ -119,10 +123,13 @@ from govoplan_files.backend.storage.connector_browse import (
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_imports import ConnectorImportError, ConnectorImportUnsupported, read_connector_file
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_effective_endpoint_url,
|
||||
reject_api_controlled_deployment_references,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profile_store import (
|
||||
connector_profile_from_row,
|
||||
create_connector_profile_row,
|
||||
deactivate_connector_profile_row,
|
||||
get_connector_profile_row,
|
||||
list_database_connector_profiles,
|
||||
update_connector_profile_row,
|
||||
@@ -570,11 +577,48 @@ def _discovery_profile_from_payload(
|
||||
password_value=credentials.password,
|
||||
token_value=credentials.token,
|
||||
capabilities=("browse",),
|
||||
metadata=payload.metadata,
|
||||
# Discovery metadata is untrusted API input and must not be able to
|
||||
# replace the candidate endpoint or inject a static/fake listing.
|
||||
metadata={},
|
||||
source_kind="discovery",
|
||||
)
|
||||
|
||||
|
||||
def _audit_connector_discovery_attempt(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
provider: str,
|
||||
endpoint_url: str,
|
||||
base_path: str | None,
|
||||
) -> None:
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="files.connector.discovery_attempted",
|
||||
object_type="connector_endpoint",
|
||||
object_id=provider,
|
||||
details={
|
||||
"provider": provider,
|
||||
"endpoint_origin": _redacted_connector_url(endpoint_url),
|
||||
"base_path": base_path,
|
||||
},
|
||||
commit=True,
|
||||
)
|
||||
|
||||
|
||||
def _redacted_connector_url(value: str) -> str:
|
||||
try:
|
||||
parsed = urlsplit(value)
|
||||
hostname = parsed.hostname or ""
|
||||
if ":" in hostname:
|
||||
hostname = f"[{hostname}]"
|
||||
netloc = f"{hostname}:{parsed.port}" if parsed.port is not None else hostname
|
||||
return urlunsplit((parsed.scheme, netloc, "", "", ""))
|
||||
except ValueError:
|
||||
return "<invalid connector URL>"
|
||||
|
||||
|
||||
def _visible_connector_profile(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
@@ -767,7 +811,11 @@ def _download_connector_payload(
|
||||
provider=profile.provider,
|
||||
external_id=f"{payload.library_id}:{source_path}",
|
||||
external_path=source_path,
|
||||
external_url=profile.endpoint_url,
|
||||
external_url=connector_effective_endpoint_url(
|
||||
provider=profile.provider,
|
||||
endpoint_url=profile.endpoint_url,
|
||||
metadata=profile.metadata,
|
||||
),
|
||||
operation=operation,
|
||||
),
|
||||
profile.policy_sources,
|
||||
@@ -802,6 +850,14 @@ def _download_connector_payload(
|
||||
return source_path, downloaded, metadata or {}
|
||||
|
||||
|
||||
def _connector_browse_next_token(items: list[Any]) -> str | None:
|
||||
for item in items:
|
||||
token = item.metadata.get("next_continuation_token") if hasattr(item, "metadata") else None
|
||||
if token:
|
||||
return str(token)
|
||||
return None
|
||||
|
||||
|
||||
def _connector_audit_details(asset: FileAsset, version: FileVersion, blob: FileBlob, *, operation: str) -> dict[str, object] | None:
|
||||
metadata = asset.metadata_ or {}
|
||||
provenance = source_provenance_from_metadata(metadata)
|
||||
@@ -937,38 +993,8 @@ def _asset_response(session: Session, asset: FileAsset, *, include_shares: bool
|
||||
)
|
||||
|
||||
|
||||
def _asset_list_response(session: Session, assets: list[FileAsset], *, include_shares: bool = False) -> list[FileAssetResponse]:
|
||||
if not assets:
|
||||
return []
|
||||
|
||||
asset_ids = [asset.id for asset in assets]
|
||||
version_ids = [asset.current_version_id for asset in assets if asset.current_version_id]
|
||||
if len(version_ids) != len(assets):
|
||||
raise FileStorageError("File has no current version")
|
||||
|
||||
version_blob_rows: list[tuple[FileVersion, FileBlob]] = []
|
||||
for chunk in _chunks(version_ids):
|
||||
version_blob_rows.extend(
|
||||
session.query(FileVersion, FileBlob)
|
||||
.join(FileBlob, FileBlob.id == FileVersion.blob_id)
|
||||
.filter(FileVersion.id.in_(chunk))
|
||||
.all()
|
||||
)
|
||||
versions_by_id = {version.id: version for version, _blob in version_blob_rows}
|
||||
blobs_by_version_id = {version.id: blob for version, blob in version_blob_rows}
|
||||
|
||||
shares_by_asset_id: dict[str, list[FileShareResponse]] = {asset_id: [] for asset_id in asset_ids}
|
||||
if include_shares:
|
||||
for chunk in _chunks(asset_ids):
|
||||
share_rows = (
|
||||
session.query(FileShare)
|
||||
.filter(FileShare.file_asset_id.in_(chunk))
|
||||
.order_by(FileShare.created_at.desc())
|
||||
.all()
|
||||
)
|
||||
for row in share_rows:
|
||||
shares_by_asset_id.setdefault(row.file_asset_id, []).append(
|
||||
FileShareResponse(
|
||||
def _file_share_response(row: FileShare) -> FileShareResponse:
|
||||
return FileShareResponse(
|
||||
id=row.id,
|
||||
target_type=row.target_type,
|
||||
target_id=row.target_id,
|
||||
@@ -976,27 +1002,62 @@ def _asset_list_response(session: Session, assets: list[FileAsset], *, include_s
|
||||
created_at=row.created_at.isoformat(),
|
||||
revoked_at=row.revoked_at.isoformat() if row.revoked_at else None,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _asset_versions_and_blobs(session: Session, assets: list[FileAsset]) -> dict[str, tuple[FileVersion, FileBlob]]:
|
||||
version_ids = [asset.current_version_id for asset in assets if asset.current_version_id]
|
||||
if len(version_ids) != len(assets):
|
||||
raise FileStorageError("File has no current version")
|
||||
rows: list[tuple[FileVersion, FileBlob]] = []
|
||||
for chunk in _chunks(version_ids):
|
||||
rows.extend(
|
||||
session.query(FileVersion, FileBlob)
|
||||
.join(FileBlob, FileBlob.id == FileVersion.blob_id)
|
||||
.filter(FileVersion.id.in_(chunk))
|
||||
.all()
|
||||
)
|
||||
return {version.id: (version, blob) for version, blob in rows}
|
||||
|
||||
|
||||
def _asset_shares_by_id(session: Session, asset_ids: list[str], *, include_shares: bool) -> dict[str, list[FileShareResponse]]:
|
||||
shares_by_asset_id: dict[str, list[FileShareResponse]] = {asset_id: [] for asset_id in asset_ids}
|
||||
if not include_shares:
|
||||
return shares_by_asset_id
|
||||
for chunk in _chunks(asset_ids):
|
||||
rows = (
|
||||
session.query(FileShare)
|
||||
.filter(FileShare.file_asset_id.in_(chunk))
|
||||
.order_by(FileShare.created_at.desc())
|
||||
.all()
|
||||
)
|
||||
for row in rows:
|
||||
shares_by_asset_id.setdefault(row.file_asset_id, []).append(_file_share_response(row))
|
||||
return shares_by_asset_id
|
||||
|
||||
|
||||
def _sent_campaign_asset_ids(session: Session, asset_ids: list[str]) -> set[str]:
|
||||
sent_asset_ids: set[str] = set()
|
||||
for chunk in _chunks(asset_ids):
|
||||
sent_rows = (
|
||||
rows = (
|
||||
session.query(CampaignAttachmentUse.file_asset_id)
|
||||
.filter(CampaignAttachmentUse.file_asset_id.in_(chunk), CampaignAttachmentUse.use_stage == "sent")
|
||||
.distinct()
|
||||
.all()
|
||||
)
|
||||
sent_asset_ids.update(row[0] for row in sent_rows)
|
||||
sent_asset_ids.update(row[0] for row in rows)
|
||||
return sent_asset_ids
|
||||
|
||||
responses: list[FileAssetResponse] = []
|
||||
for asset in assets:
|
||||
version = versions_by_id.get(asset.current_version_id or "")
|
||||
blob = blobs_by_version_id.get(asset.current_version_id or "")
|
||||
if not version or not blob:
|
||||
raise FileStorageError("File version not found")
|
||||
|
||||
def _loaded_asset_response(
|
||||
asset: FileAsset,
|
||||
*,
|
||||
version: FileVersion,
|
||||
blob: FileBlob,
|
||||
shares: list[FileShareResponse],
|
||||
sent_asset_ids: set[str],
|
||||
) -> FileAssetResponse:
|
||||
metadata = asset.metadata_ or {}
|
||||
responses.append(
|
||||
FileAssetResponse(
|
||||
return FileAssetResponse(
|
||||
id=asset.id,
|
||||
tenant_id=asset.tenant_id,
|
||||
owner_type=asset.owner_type,
|
||||
@@ -1015,7 +1076,32 @@ def _asset_list_response(session: Session, assets: list[FileAsset], *, include_s
|
||||
metadata=metadata,
|
||||
source_provenance=source_provenance_from_metadata(metadata),
|
||||
source_revision=source_revision_from_metadata(metadata),
|
||||
shares=shares,
|
||||
)
|
||||
|
||||
|
||||
def _asset_list_response(session: Session, assets: list[FileAsset], *, include_shares: bool = False) -> list[FileAssetResponse]:
|
||||
if not assets:
|
||||
return []
|
||||
|
||||
asset_ids = [asset.id for asset in assets]
|
||||
versions_and_blobs = _asset_versions_and_blobs(session, assets)
|
||||
shares_by_asset_id = _asset_shares_by_id(session, asset_ids, include_shares=include_shares)
|
||||
sent_asset_ids = _sent_campaign_asset_ids(session, asset_ids)
|
||||
|
||||
responses: list[FileAssetResponse] = []
|
||||
for asset in assets:
|
||||
version_and_blob = versions_and_blobs.get(asset.current_version_id or "")
|
||||
if version_and_blob is None:
|
||||
raise FileStorageError("File version not found")
|
||||
version, blob = version_and_blob
|
||||
responses.append(
|
||||
_loaded_asset_response(
|
||||
asset,
|
||||
version=version,
|
||||
blob=blob,
|
||||
shares=shares_by_asset_id.get(asset.id, []),
|
||||
sent_asset_ids=sent_asset_ids,
|
||||
)
|
||||
)
|
||||
return responses
|
||||
@@ -1099,7 +1185,11 @@ def _connector_space_policy_decision(
|
||||
provider=profile.provider,
|
||||
external_id=external_id,
|
||||
external_path=browse_path,
|
||||
external_url=profile.endpoint_url,
|
||||
external_url=connector_effective_endpoint_url(
|
||||
provider=profile.provider,
|
||||
endpoint_url=profile.endpoint_url,
|
||||
metadata=profile.metadata,
|
||||
),
|
||||
operation=operation,
|
||||
),
|
||||
profile.policy_sources,
|
||||
@@ -1441,6 +1531,104 @@ def _entry_matches_delta_scope(
|
||||
)
|
||||
|
||||
|
||||
def _decode_files_delta_watermark(since: str) -> int:
|
||||
try:
|
||||
return decode_sequence_watermark(since)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc)) from exc
|
||||
|
||||
|
||||
def _changed_delta_resource_ids(entries: list[ChangeSequenceEntry]) -> tuple[list[str], list[str]]:
|
||||
file_ids = list(dict.fromkeys(entry.resource_id for entry in entries if entry.resource_type == "file"))
|
||||
folder_ids = list(dict.fromkeys(entry.resource_id for entry in entries if entry.resource_type == "folder"))
|
||||
return file_ids, folder_ids
|
||||
|
||||
|
||||
def _visible_assets_for_delta(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
changed_file_ids: list[str],
|
||||
) -> dict[str, FileAsset]:
|
||||
return {
|
||||
asset.id: asset
|
||||
for asset in list_assets_for_user(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
is_admin=_is_admin(principal),
|
||||
)
|
||||
if asset.id in changed_file_ids
|
||||
}
|
||||
|
||||
|
||||
def _changed_visible_folders_for_delta(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
path_prefix: str | None,
|
||||
changed_folder_ids: list[str],
|
||||
) -> dict[str, FileFolder]:
|
||||
return {
|
||||
folder.id: folder
|
||||
for folder in _visible_folders_for_delta(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
path_prefix=path_prefix,
|
||||
)
|
||||
if folder.id in changed_folder_ids
|
||||
}
|
||||
|
||||
|
||||
def _deleted_delta_items(
|
||||
session: Session,
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
entries: list[ChangeSequenceEntry],
|
||||
visible_assets: dict[str, FileAsset],
|
||||
visible_folders: dict[str, FileFolder],
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
campaign_id: str | None,
|
||||
path_prefix: str | None,
|
||||
) -> list[DeltaDeletedItem]:
|
||||
deleted: dict[tuple[str, str], DeltaDeletedItem] = {}
|
||||
for entry in entries:
|
||||
if entry.resource_type == "file" and entry.resource_id in visible_assets:
|
||||
continue
|
||||
if entry.resource_type == "folder" and entry.resource_id in visible_folders:
|
||||
continue
|
||||
if not _entry_matches_delta_scope(
|
||||
session,
|
||||
principal,
|
||||
entry,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
):
|
||||
continue
|
||||
deleted[(entry.resource_type, entry.resource_id)] = DeltaDeletedItem(
|
||||
id=entry.resource_id,
|
||||
resource_type=entry.resource_type,
|
||||
revision=encode_sequence_watermark(entry.id),
|
||||
deleted_at=entry.created_at if entry.operation == "deleted" else None,
|
||||
)
|
||||
return list(deleted.values())
|
||||
|
||||
|
||||
def _files_delta_response(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -1452,10 +1640,7 @@ def _files_delta_response(
|
||||
since: str,
|
||||
limit: int,
|
||||
) -> FileDeltaResponse:
|
||||
try:
|
||||
since_sequence = decode_sequence_watermark(since)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc)) from exc
|
||||
since_sequence = _decode_files_delta_watermark(since)
|
||||
if sequence_watermark_is_expired(
|
||||
session,
|
||||
since=since_sequence,
|
||||
@@ -1483,65 +1668,41 @@ def _files_delta_response(
|
||||
has_more = len(entries_plus_one) > limit
|
||||
entries = entries_plus_one[:limit]
|
||||
|
||||
changed_file_ids = list(dict.fromkeys(entry.resource_id for entry in entries if entry.resource_type == "file"))
|
||||
changed_folder_ids = list(dict.fromkeys(entry.resource_id for entry in entries if entry.resource_type == "folder"))
|
||||
|
||||
visible_assets = {
|
||||
asset.id: asset
|
||||
for asset in list_assets_for_user(
|
||||
changed_file_ids, changed_folder_ids = _changed_delta_resource_ids(entries)
|
||||
visible_assets = _visible_assets_for_delta(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=principal.user.id,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
is_admin=_is_admin(principal),
|
||||
changed_file_ids=changed_file_ids,
|
||||
)
|
||||
if asset.id in changed_file_ids
|
||||
}
|
||||
visible_folders = {
|
||||
folder.id: folder
|
||||
for folder in _visible_folders_for_delta(
|
||||
visible_folders = _changed_visible_folders_for_delta(
|
||||
session,
|
||||
principal=principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
path_prefix=path_prefix,
|
||||
changed_folder_ids=changed_folder_ids,
|
||||
)
|
||||
if folder.id in changed_folder_ids
|
||||
}
|
||||
|
||||
deleted: dict[tuple[str, str], DeltaDeletedItem] = {}
|
||||
for entry in entries:
|
||||
is_visible = (
|
||||
(entry.resource_type == "file" and entry.resource_id in visible_assets)
|
||||
or (entry.resource_type == "folder" and entry.resource_id in visible_folders)
|
||||
)
|
||||
if is_visible:
|
||||
continue
|
||||
if not _entry_matches_delta_scope(
|
||||
deleted = _deleted_delta_items(
|
||||
session,
|
||||
principal,
|
||||
entry,
|
||||
principal=principal,
|
||||
entries=entries,
|
||||
visible_assets=visible_assets,
|
||||
visible_folders=visible_folders,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
campaign_id=campaign_id,
|
||||
path_prefix=path_prefix,
|
||||
):
|
||||
continue
|
||||
deleted[(entry.resource_type, entry.resource_id)] = DeltaDeletedItem(
|
||||
id=entry.resource_id,
|
||||
resource_type=entry.resource_type,
|
||||
revision=encode_sequence_watermark(entry.id),
|
||||
deleted_at=entry.created_at if entry.operation == "deleted" else None,
|
||||
)
|
||||
|
||||
watermark = encode_sequence_watermark(entries[-1].id) if has_more and entries else _files_delta_watermark(session, principal.tenant_id)
|
||||
return FileDeltaResponse(
|
||||
files=_asset_list_response(session, list(visible_assets.values()), include_shares=True),
|
||||
folders=[_folder_response(folder) for folder in visible_folders.values()],
|
||||
deleted=list(deleted.values()),
|
||||
deleted=deleted,
|
||||
watermark=watermark,
|
||||
has_more=has_more,
|
||||
full=False,
|
||||
@@ -2148,6 +2309,148 @@ def _full_file_connector_settings_delta_response(
|
||||
)
|
||||
|
||||
|
||||
def _changed_file_connector_setting_ids(entries: list[ChangeSequenceEntry]) -> tuple[set[str], set[str], set[str], bool]:
|
||||
changed_profile_ids = {
|
||||
entry.resource_id
|
||||
for entry in entries
|
||||
if entry.collection == FILES_CONNECTOR_PROFILES_COLLECTION and entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
|
||||
}
|
||||
changed_credential_ids = {
|
||||
entry.resource_id
|
||||
for entry in entries
|
||||
if entry.collection == FILES_CONNECTOR_CREDENTIALS_COLLECTION and entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
|
||||
}
|
||||
changed_space_ids = {
|
||||
entry.resource_id
|
||||
for entry in entries
|
||||
if entry.collection == FILES_CONNECTOR_SPACES_COLLECTION and entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
|
||||
}
|
||||
policy_changed = any(entry.collection == FILES_CONNECTOR_POLICIES_COLLECTION for entry in entries)
|
||||
return changed_profile_ids, changed_credential_ids, changed_space_ids, policy_changed
|
||||
|
||||
|
||||
def _file_connector_settings_changed_sections(
|
||||
*,
|
||||
changed_profile_ids: set[str],
|
||||
changed_credential_ids: set[str],
|
||||
changed_space_ids: set[str],
|
||||
policy_changed: bool,
|
||||
profiles: list[ConnectorProfile],
|
||||
) -> list[str]:
|
||||
changed_sections = []
|
||||
if changed_profile_ids or any(profile.credential_profile_id and profile.credential_profile_id in changed_credential_ids for profile in profiles):
|
||||
changed_sections.append("profiles")
|
||||
if changed_credential_ids:
|
||||
changed_sections.append("credentials")
|
||||
if changed_space_ids:
|
||||
changed_sections.append("spaces")
|
||||
if policy_changed:
|
||||
changed_sections.append("policy")
|
||||
return changed_sections
|
||||
|
||||
|
||||
def _file_connector_settings_deleted_items(
|
||||
entries: list[ChangeSequenceEntry],
|
||||
*,
|
||||
visible_profiles: dict[str, ConnectorProfile],
|
||||
visible_credentials: dict[str, ConnectorCredential],
|
||||
visible_spaces: dict[str, FileConnectorSpace],
|
||||
) -> list[DeltaDeletedItem]:
|
||||
return [
|
||||
_connector_deleted_item(entry)
|
||||
for entry in entries
|
||||
if (
|
||||
entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
|
||||
and entry.resource_id not in visible_profiles
|
||||
)
|
||||
or (
|
||||
entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
|
||||
and entry.resource_id not in visible_credentials
|
||||
)
|
||||
or (
|
||||
entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
|
||||
and entry.resource_id not in visible_spaces
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def _incremental_file_connector_settings_delta_response(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
entries: list[ChangeSequenceEntry],
|
||||
has_more: bool,
|
||||
scope_type: str,
|
||||
scope_id: str | None,
|
||||
provider: str | None,
|
||||
campaign_id: str | None,
|
||||
include_disabled: bool,
|
||||
include_inactive: bool,
|
||||
owner_type: Literal["user", "group"] | None,
|
||||
owner_id: str | None,
|
||||
) -> FileConnectorSettingsDeltaResponse:
|
||||
changed_profile_ids, changed_credential_ids, changed_space_ids, policy_changed = _changed_file_connector_setting_ids(entries)
|
||||
profiles = _visible_connector_profiles(
|
||||
session,
|
||||
principal,
|
||||
provider=provider,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled and _can_read_disabled_connector_profiles(principal),
|
||||
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
|
||||
include_effective_policy=False,
|
||||
)
|
||||
visible_profiles = {
|
||||
profile.id: profile
|
||||
for profile in profiles
|
||||
if profile.id in changed_profile_ids or (profile.credential_profile_id and profile.credential_profile_id in changed_credential_ids)
|
||||
}
|
||||
credentials = _visible_connector_credentials(
|
||||
session,
|
||||
principal,
|
||||
provider=provider,
|
||||
include_disabled=include_disabled,
|
||||
)
|
||||
visible_credentials = {
|
||||
credential.id: credential
|
||||
for credential in credentials
|
||||
if credential.id in changed_credential_ids
|
||||
}
|
||||
spaces = _visible_connector_spaces(
|
||||
session,
|
||||
principal,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
include_inactive=include_inactive,
|
||||
)
|
||||
visible_spaces = {
|
||||
space.id: space
|
||||
for space in spaces
|
||||
if space.id in changed_space_ids
|
||||
}
|
||||
return FileConnectorSettingsDeltaResponse(
|
||||
profiles=[FileConnectorProfileResponse(**profile.to_response()) for profile in visible_profiles.values()],
|
||||
credentials=[FileConnectorCredentialResponse(**credential.to_response()) for credential in visible_credentials.values()],
|
||||
spaces=[_connector_space_response(space) for space in visible_spaces.values()],
|
||||
policy=FileConnectorPolicyResponse(**connector_policy_response(session, tenant_id=principal.tenant_id, scope_type=scope_type, scope_id=scope_id)) if policy_changed else None,
|
||||
changed_sections=_file_connector_settings_changed_sections(
|
||||
changed_profile_ids=changed_profile_ids,
|
||||
changed_credential_ids=changed_credential_ids,
|
||||
changed_space_ids=changed_space_ids,
|
||||
policy_changed=policy_changed,
|
||||
profiles=profiles,
|
||||
),
|
||||
deleted=_file_connector_settings_deleted_items(
|
||||
entries,
|
||||
visible_profiles=visible_profiles,
|
||||
visible_credentials=visible_credentials,
|
||||
visible_spaces=visible_spaces,
|
||||
),
|
||||
watermark=_file_connector_settings_response_watermark(session, tenant_id=principal.tenant_id, entries=entries, has_more=has_more),
|
||||
has_more=has_more,
|
||||
full=False,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/connectors/settings/delta", response_model=FileConnectorSettingsDeltaResponse)
|
||||
def connector_settings_delta(
|
||||
scope_type: str = Query(default="tenant"),
|
||||
@@ -2193,94 +2496,19 @@ def connector_settings_delta(
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
)
|
||||
changed_profile_ids = {
|
||||
entry.resource_id
|
||||
for entry in entries
|
||||
if entry.collection == FILES_CONNECTOR_PROFILES_COLLECTION and entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
|
||||
}
|
||||
changed_credential_ids = {
|
||||
entry.resource_id
|
||||
for entry in entries
|
||||
if entry.collection == FILES_CONNECTOR_CREDENTIALS_COLLECTION and entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
|
||||
}
|
||||
changed_space_ids = {
|
||||
entry.resource_id
|
||||
for entry in entries
|
||||
if entry.collection == FILES_CONNECTOR_SPACES_COLLECTION and entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
|
||||
}
|
||||
policy_changed = any(entry.collection == FILES_CONNECTOR_POLICIES_COLLECTION for entry in entries)
|
||||
profiles = _visible_connector_profiles(
|
||||
return _incremental_file_connector_settings_delta_response(
|
||||
session,
|
||||
principal,
|
||||
entries=entries,
|
||||
has_more=has_more,
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
provider=provider,
|
||||
campaign_id=campaign_id,
|
||||
include_disabled=include_disabled and _can_read_disabled_connector_profiles(principal),
|
||||
include_admin_scopes=_can_read_disabled_connector_profiles(principal),
|
||||
include_effective_policy=False,
|
||||
)
|
||||
visible_profiles = {
|
||||
profile.id: profile
|
||||
for profile in profiles
|
||||
if profile.id in changed_profile_ids or (profile.credential_profile_id and profile.credential_profile_id in changed_credential_ids)
|
||||
}
|
||||
credentials = _visible_connector_credentials(
|
||||
session,
|
||||
principal,
|
||||
provider=provider,
|
||||
include_disabled=include_disabled,
|
||||
)
|
||||
visible_credentials = {
|
||||
credential.id: credential
|
||||
for credential in credentials
|
||||
if credential.id in changed_credential_ids
|
||||
}
|
||||
spaces = _visible_connector_spaces(
|
||||
session,
|
||||
principal,
|
||||
include_inactive=include_inactive,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
include_inactive=include_inactive,
|
||||
)
|
||||
visible_spaces = {
|
||||
space.id: space
|
||||
for space in spaces
|
||||
if space.id in changed_space_ids
|
||||
}
|
||||
changed_sections = []
|
||||
if changed_profile_ids or any(profile.credential_profile_id and profile.credential_profile_id in changed_credential_ids for profile in profiles):
|
||||
changed_sections.append("profiles")
|
||||
if changed_credential_ids:
|
||||
changed_sections.append("credentials")
|
||||
if changed_space_ids:
|
||||
changed_sections.append("spaces")
|
||||
if policy_changed:
|
||||
changed_sections.append("policy")
|
||||
deleted = [
|
||||
_connector_deleted_item(entry)
|
||||
for entry in entries
|
||||
if (
|
||||
entry.resource_type == FILES_CONNECTOR_PROFILE_RESOURCE
|
||||
and entry.resource_id not in visible_profiles
|
||||
)
|
||||
or (
|
||||
entry.resource_type == FILES_CONNECTOR_CREDENTIAL_RESOURCE
|
||||
and entry.resource_id not in visible_credentials
|
||||
)
|
||||
or (
|
||||
entry.resource_type == FILES_CONNECTOR_SPACE_RESOURCE
|
||||
and entry.resource_id not in visible_spaces
|
||||
)
|
||||
]
|
||||
return FileConnectorSettingsDeltaResponse(
|
||||
profiles=[FileConnectorProfileResponse(**profile.to_response()) for profile in visible_profiles.values()],
|
||||
credentials=[FileConnectorCredentialResponse(**credential.to_response()) for credential in visible_credentials.values()],
|
||||
spaces=[_connector_space_response(space) for space in visible_spaces.values()],
|
||||
policy=FileConnectorPolicyResponse(**connector_policy_response(session, tenant_id=principal.tenant_id, scope_type=scope_type, scope_id=scope_id)) if policy_changed else None,
|
||||
changed_sections=changed_sections,
|
||||
deleted=deleted,
|
||||
watermark=_file_connector_settings_response_watermark(session, tenant_id=principal.tenant_id, entries=entries, has_more=has_more),
|
||||
has_more=has_more,
|
||||
full=False,
|
||||
)
|
||||
except (FileStorageError, ValueError, json.JSONDecodeError) as exc:
|
||||
raise _http_error(exc) from exc
|
||||
@@ -2297,8 +2525,18 @@ def list_connector_providers(
|
||||
@router.post("/connectors/discover", response_model=FileConnectorDiscoveryResponse)
|
||||
def discover_connector_endpoint(
|
||||
payload: FileConnectorDiscoveryRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_any_scope("files:file:admin", "system:settings:write", "admin:settings:write")),
|
||||
):
|
||||
try:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=payload.credentials.password_env,
|
||||
token_env=payload.credentials.token_env,
|
||||
secret_ref=payload.credentials.secret_ref,
|
||||
metadata=payload.metadata,
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
if payload.provider not in {"webdav", "nextcloud"}:
|
||||
return FileConnectorDiscoveryResponse(
|
||||
provider=payload.provider,
|
||||
@@ -2311,7 +2549,28 @@ def discover_connector_endpoint(
|
||||
for endpoint_url in _webdav_discovery_candidates(payload):
|
||||
profile = _discovery_profile_from_payload(payload, endpoint_url, principal=principal)
|
||||
try:
|
||||
_ensure_connector_configuration_allowed(
|
||||
session,
|
||||
principal,
|
||||
connector_id=None,
|
||||
credential_id=None,
|
||||
provider=profile.provider,
|
||||
endpoint_url=endpoint_url,
|
||||
base_path=profile.base_path,
|
||||
scope_type="tenant",
|
||||
scope_id=principal.tenant_id,
|
||||
operation="discover",
|
||||
)
|
||||
_audit_connector_discovery_attempt(
|
||||
session,
|
||||
principal,
|
||||
provider=profile.provider,
|
||||
endpoint_url=endpoint_url,
|
||||
base_path=profile.base_path,
|
||||
)
|
||||
browse_connector_profile(profile, path=payload.base_path or "")
|
||||
except ConnectorPolicyDenied as exc:
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except (ConnectorBrowseError, ConnectorBrowseUnsupported, OSError, ValueError, json.JSONDecodeError) as exc:
|
||||
message = str(exc)
|
||||
if "credentials were rejected" in message.casefold():
|
||||
@@ -2324,7 +2583,7 @@ def discover_connector_endpoint(
|
||||
status="credentials_rejected",
|
||||
message="The endpoint was found, but login failed with these credentials.",
|
||||
candidates=candidates,
|
||||
metadata={**payload.metadata, "discovered_by": "webdav-auth-challenge"},
|
||||
metadata={"discovered_by": "webdav-auth-challenge"},
|
||||
)
|
||||
candidates.append({"endpoint_url": endpoint_url, "status": "found", "message": "The endpoint exists, but credentials are required or were rejected."})
|
||||
return FileConnectorDiscoveryResponse(
|
||||
@@ -2334,7 +2593,7 @@ def discover_connector_endpoint(
|
||||
status="found",
|
||||
message="The endpoint was found. Add working credentials before saving or testing the connection.",
|
||||
candidates=candidates,
|
||||
metadata={**payload.metadata, "discovered_by": "webdav-auth-challenge"},
|
||||
metadata={"discovered_by": "webdav-auth-challenge"},
|
||||
)
|
||||
candidates.append({"endpoint_url": endpoint_url, "status": "failed", "message": message})
|
||||
continue
|
||||
@@ -2348,7 +2607,7 @@ def discover_connector_endpoint(
|
||||
status=status_value,
|
||||
message=message,
|
||||
candidates=candidates,
|
||||
metadata={**payload.metadata, "discovered_by": "webdav-propfind"},
|
||||
metadata={"discovered_by": "webdav-propfind"},
|
||||
)
|
||||
return FileConnectorDiscoveryResponse(
|
||||
provider=payload.provider,
|
||||
@@ -2600,7 +2859,14 @@ def deactivate_connector_credential(
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
_require_connector_credential_write(principal, row.scope_type)
|
||||
try:
|
||||
deactivate_connector_credential_row(session, row, user_id=principal.user.id)
|
||||
deletion = delete_connector_credential_row(
|
||||
session,
|
||||
row,
|
||||
deletion_reason="api_delete",
|
||||
user_id=principal.user.id,
|
||||
api_key_id=principal.api_key.id if principal.api_key else None,
|
||||
)
|
||||
if deletion.changed:
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_CREDENTIALS_COLLECTION,
|
||||
@@ -2611,12 +2877,31 @@ def deactivate_connector_credential(
|
||||
tenant_id=row.tenant_id,
|
||||
payload={"scope_type": row.scope_type, "scope_id": row.scope_id, "provider": row.provider},
|
||||
)
|
||||
for profile in deletion.affected_profiles:
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
resource_type=FILES_CONNECTOR_PROFILE_RESOURCE,
|
||||
resource_id=profile.id,
|
||||
operation="updated",
|
||||
principal=principal,
|
||||
tenant_id=profile.tenant_id,
|
||||
payload={
|
||||
"scope_type": profile.scope_type,
|
||||
"scope_id": profile.scope_id,
|
||||
"provider": profile.provider,
|
||||
"reason": "credential_deleted",
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return _connector_credential_response(row)
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
except Exception:
|
||||
session.rollback()
|
||||
raise
|
||||
|
||||
|
||||
@router.get("/connectors/profiles", response_model=FileConnectorProfilesResponse)
|
||||
@@ -2664,8 +2949,12 @@ def create_connector_profile(
|
||||
principal,
|
||||
connector_id=payload.id,
|
||||
credential_id=payload.credential_profile_id,
|
||||
provider=payload.provider,
|
||||
endpoint_url=connector_effective_endpoint_url(
|
||||
provider=payload.provider,
|
||||
endpoint_url=payload.endpoint_url,
|
||||
metadata=payload.metadata,
|
||||
),
|
||||
base_path=payload.base_path,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=payload.scope_id,
|
||||
@@ -2819,6 +3108,7 @@ def browse_connector_profile_items(
|
||||
profile_id: str,
|
||||
path: str | None = None,
|
||||
library_id: str | None = None,
|
||||
continuation_token: str | None = None,
|
||||
campaign_id: str | None = None,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(require_any_scope("files:file:read", "files:file:upload", "files:file:download", "files:file:admin", "system:settings:read", "admin:settings:read")),
|
||||
@@ -2832,14 +3122,18 @@ def browse_connector_profile_items(
|
||||
credential_id=profile.credential_profile_id,
|
||||
provider=profile.provider,
|
||||
external_path=browse_path,
|
||||
external_url=profile.endpoint_url,
|
||||
external_url=connector_effective_endpoint_url(
|
||||
provider=profile.provider,
|
||||
endpoint_url=profile.endpoint_url,
|
||||
metadata=profile.metadata,
|
||||
),
|
||||
operation="browse",
|
||||
),
|
||||
profile.policy_sources,
|
||||
)
|
||||
if not decision.allowed:
|
||||
raise ConnectorPolicyDenied(decision)
|
||||
items = browse_connector_profile(profile, path=browse_path, library_id=library_id)
|
||||
items = browse_connector_profile(profile, path=browse_path, library_id=library_id, continuation_token=continuation_token)
|
||||
except ConnectorPolicyDenied as exc:
|
||||
raise _connector_policy_error(exc) from exc
|
||||
except ConnectorBrowseUnsupported as exc:
|
||||
@@ -2851,6 +3145,8 @@ def browse_connector_profile_items(
|
||||
provider=profile.provider,
|
||||
path=browse_path,
|
||||
library_id=library_id,
|
||||
next_continuation_token=_connector_browse_next_token(items),
|
||||
has_more=any(bool(item.metadata.get("listing_truncated")) for item in items),
|
||||
decision=decision.to_dict(),
|
||||
items=[FileConnectorBrowseItem(**item.to_response()) for item in items],
|
||||
)
|
||||
@@ -2906,8 +3202,12 @@ def update_connector_profile(
|
||||
principal,
|
||||
connector_id=row.id,
|
||||
credential_id=credential_profile_id,
|
||||
provider=provider,
|
||||
endpoint_url=connector_effective_endpoint_url(
|
||||
provider=provider,
|
||||
endpoint_url=payload.endpoint_url if payload.endpoint_url is not None else row.endpoint_url,
|
||||
metadata=payload.metadata if payload.metadata is not None else row.metadata_,
|
||||
),
|
||||
base_path=payload.base_path if payload.base_path is not None else row.base_path,
|
||||
scope_type=row.scope_type,
|
||||
scope_id=row.scope_id,
|
||||
@@ -2977,7 +3277,14 @@ def deactivate_connector_profile(
|
||||
raise _http_error(exc, not_found=True) from exc
|
||||
_require_connector_profile_write(principal, row.scope_type)
|
||||
try:
|
||||
deactivate_connector_profile_row(session, row, user_id=principal.user.id)
|
||||
changed = delete_connector_profile_row(
|
||||
session,
|
||||
row,
|
||||
deletion_reason="api_delete",
|
||||
user_id=principal.user.id,
|
||||
api_key_id=principal.api_key.id if principal.api_key else None,
|
||||
)
|
||||
if changed:
|
||||
_record_connector_settings_change(
|
||||
session,
|
||||
collection=FILES_CONNECTOR_PROFILES_COLLECTION,
|
||||
@@ -2994,6 +3301,9 @@ def deactivate_connector_profile(
|
||||
except FileStorageError as exc:
|
||||
session.rollback()
|
||||
raise _http_error(exc) from exc
|
||||
except Exception:
|
||||
session.rollback()
|
||||
raise
|
||||
|
||||
|
||||
@router.get("/{file_id}", response_model=FileAssetResponse)
|
||||
@@ -3230,7 +3540,7 @@ def download_archive(
|
||||
get_asset_for_user(session, tenant_id=principal.tenant_id, user_id=principal.user.id, asset_id=file_id, is_admin=_is_admin(principal))
|
||||
for file_id in payload.file_ids
|
||||
]
|
||||
tmp = tempfile.NamedTemporaryFile(prefix="multimailer-files-", suffix=".zip", delete=False)
|
||||
tmp = tempfile.NamedTemporaryFile(prefix="govoplan-files-", suffix=".zip", delete=False)
|
||||
tmp_path = tmp.name
|
||||
tmp.close()
|
||||
try:
|
||||
|
||||
@@ -1,36 +1,10 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from govoplan_core.core.runtime import ModuleRuntimeState
|
||||
|
||||
_runtime_registry: object | None = None
|
||||
_runtime_settings: object | None = None
|
||||
_runtime = ModuleRuntimeState("Files")
|
||||
|
||||
|
||||
def configure_runtime(*, registry: object | None = None, settings: object | None = None) -> None:
|
||||
global _runtime_registry, _runtime_settings
|
||||
if registry is not None:
|
||||
_runtime_registry = registry
|
||||
if settings is not None:
|
||||
_runtime_settings = settings
|
||||
|
||||
|
||||
def get_registry() -> object | None:
|
||||
return _runtime_registry
|
||||
|
||||
|
||||
def get_settings() -> object:
|
||||
if _runtime_settings is not None:
|
||||
return _runtime_settings
|
||||
try:
|
||||
from govoplan_core.settings import settings as legacy_settings
|
||||
except ModuleNotFoundError as exc:
|
||||
raise RuntimeError("GovOPlaN Files runtime settings are not configured") from exc
|
||||
return legacy_settings
|
||||
|
||||
|
||||
class SettingsProxy:
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
return getattr(get_settings(), name)
|
||||
|
||||
|
||||
settings = SettingsProxy()
|
||||
configure_runtime = _runtime.configure_runtime
|
||||
get_registry = _runtime.get_registry
|
||||
get_settings = _runtime.get_settings
|
||||
settings = _runtime.settings
|
||||
|
||||
@@ -287,7 +287,7 @@ class FileConnectorDiscoveryCandidate(BaseModel):
|
||||
|
||||
|
||||
class FileConnectorDiscoveryRequest(BaseModel):
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"]
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"]
|
||||
endpoint_url: str
|
||||
base_path: str | None = None
|
||||
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] = "none"
|
||||
@@ -309,7 +309,7 @@ class FileConnectorDiscoveryResponse(BaseModel):
|
||||
class FileConnectorProfileCreateRequest(BaseModel):
|
||||
id: str
|
||||
label: str
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"]
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"]
|
||||
endpoint_url: str | None = None
|
||||
base_path: str | None = None
|
||||
enabled: bool = True
|
||||
@@ -325,7 +325,7 @@ class FileConnectorProfileCreateRequest(BaseModel):
|
||||
|
||||
class FileConnectorProfileUpdateRequest(BaseModel):
|
||||
label: str | None = None
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"] | None = None
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
|
||||
endpoint_url: str | None = None
|
||||
base_path: str | None = None
|
||||
enabled: bool | None = None
|
||||
@@ -342,7 +342,7 @@ class FileConnectorProfileUpdateRequest(BaseModel):
|
||||
class FileConnectorCredentialCreateRequest(BaseModel):
|
||||
id: str
|
||||
label: str
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"] | None = None
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
|
||||
enabled: bool = True
|
||||
scope_type: Literal["system", "tenant", "user", "group", "campaign"] = "tenant"
|
||||
scope_id: str | None = None
|
||||
@@ -354,7 +354,7 @@ class FileConnectorCredentialCreateRequest(BaseModel):
|
||||
|
||||
class FileConnectorCredentialUpdateRequest(BaseModel):
|
||||
label: str | None = None
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"] | None = None
|
||||
provider: Literal["seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"] | None = None
|
||||
enabled: bool | None = None
|
||||
credential_mode: Literal["none", "anonymous", "basic", "token", "secret_ref"] | None = None
|
||||
credentials: FileConnectorProfileCredentialsRequest | None = None
|
||||
@@ -404,6 +404,8 @@ class FileConnectorBrowseResponse(BaseModel):
|
||||
path: str = ""
|
||||
library_id: str | None = None
|
||||
read_only: bool = True
|
||||
next_continuation_token: str | None = None
|
||||
has_more: bool = False
|
||||
decision: dict[str, Any]
|
||||
items: list[FileConnectorBrowseItem]
|
||||
|
||||
|
||||
@@ -4,6 +4,12 @@ from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Iterable, Protocol
|
||||
|
||||
from govoplan_core.security.outbound_http import (
|
||||
OutboundHttpError,
|
||||
response_limit,
|
||||
validate_unpinned_sdk_http_url,
|
||||
)
|
||||
|
||||
from govoplan_files.backend.runtime import settings
|
||||
|
||||
|
||||
@@ -92,19 +98,29 @@ class S3StorageBackend:
|
||||
|
||||
@property
|
||||
def client(self):
|
||||
try:
|
||||
endpoint_url = validate_unpinned_sdk_http_url(
|
||||
self.endpoint_url,
|
||||
label="File storage S3 endpoint",
|
||||
)
|
||||
except OutboundHttpError as exc:
|
||||
raise StorageBackendError(str(exc)) from exc
|
||||
try:
|
||||
import boto3
|
||||
except ModuleNotFoundError as exc:
|
||||
raise StorageBackendError("boto3 is required for the S3 storage backend") from exc
|
||||
return boto3.client(
|
||||
"s3",
|
||||
endpoint_url=self.endpoint_url,
|
||||
endpoint_url=endpoint_url,
|
||||
region_name=self.region_name,
|
||||
aws_access_key_id=self.access_key_id,
|
||||
aws_secret_access_key=self.secret_access_key,
|
||||
)
|
||||
|
||||
def put_bytes(self, key: str, data: bytes, *, content_type: str | None = None) -> None:
|
||||
max_bytes = response_limit("file")
|
||||
if len(data) > max_bytes:
|
||||
raise StorageBackendError(f"Stored object exceeds the deployment limit of {max_bytes} bytes")
|
||||
kwargs = {"Bucket": self.bucket, "Key": key, "Body": data}
|
||||
if content_type:
|
||||
kwargs["ContentType"] = content_type
|
||||
@@ -113,19 +129,39 @@ class S3StorageBackend:
|
||||
def get_bytes(self, key: str) -> bytes:
|
||||
try:
|
||||
obj = self.client.get_object(Bucket=self.bucket, Key=key)
|
||||
return obj["Body"].read()
|
||||
max_bytes = response_limit("file")
|
||||
body = obj["Body"]
|
||||
try:
|
||||
_reject_declared_object_size(obj, max_bytes=max_bytes)
|
||||
data = body.read(max_bytes + 1)
|
||||
if len(data) > max_bytes:
|
||||
raise StorageBackendError(f"Stored object exceeds the deployment limit of {max_bytes} bytes")
|
||||
return data
|
||||
finally:
|
||||
if hasattr(body, "close"):
|
||||
body.close()
|
||||
except Exception as exc: # pragma: no cover - depends on S3 backend
|
||||
raise StorageBackendError(str(exc)) from exc
|
||||
|
||||
def iter_bytes(self, key: str, *, chunk_size: int = 1024 * 1024) -> Iterable[bytes]:
|
||||
try:
|
||||
obj = self.client.get_object(Bucket=self.bucket, Key=key)
|
||||
max_bytes = response_limit("file")
|
||||
body = obj["Body"]
|
||||
try:
|
||||
_reject_declared_object_size(obj, max_bytes=max_bytes)
|
||||
total = 0
|
||||
while True:
|
||||
chunk = body.read(chunk_size)
|
||||
if not chunk:
|
||||
break
|
||||
total += len(chunk)
|
||||
if total > max_bytes:
|
||||
raise StorageBackendError(f"Stored object exceeds the deployment limit of {max_bytes} bytes")
|
||||
yield chunk
|
||||
finally:
|
||||
if hasattr(body, "close"):
|
||||
body.close()
|
||||
except Exception as exc: # pragma: no cover - depends on S3 backend
|
||||
raise StorageBackendError(str(exc)) from exc
|
||||
|
||||
@@ -140,6 +176,17 @@ class S3StorageBackend:
|
||||
return False
|
||||
|
||||
|
||||
def _reject_declared_object_size(obj: object, *, max_bytes: int) -> None:
|
||||
if not isinstance(obj, dict):
|
||||
return
|
||||
try:
|
||||
declared_size = int(obj.get("ContentLength"))
|
||||
except (TypeError, ValueError):
|
||||
return
|
||||
if declared_size > max_bytes:
|
||||
raise StorageBackendError(f"Stored object exceeds the deployment limit of {max_bytes} bytes")
|
||||
|
||||
|
||||
def _fallback_roots() -> tuple[Path, ...]:
|
||||
raw = getattr(settings, "file_storage_local_fallback_roots", "") or ""
|
||||
return tuple(Path(item.strip()) for item in str(raw).split(",") if item.strip())
|
||||
|
||||
@@ -12,10 +12,11 @@ from typing import Any, Iterator
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_files.backend.db.models import FileAsset
|
||||
from govoplan_files.backend.storage.backends import StorageBackendError, get_storage_backend
|
||||
from govoplan_files.backend.db.models import FileAsset, FileBlob, FileShare, FileVersion
|
||||
from govoplan_files.backend.storage.backends import StorageBackend, StorageBackendError, get_storage_backend
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.files import current_versions_and_blobs, list_assets_for_user
|
||||
from govoplan_files.backend.storage.files import current_versions_and_blobs, get_asset_for_user, list_assets_for_user, share_files
|
||||
from govoplan_files.backend.storage.access import ensure_owner_access
|
||||
from govoplan_files.backend.storage.paths import normalize_folder, normalize_logical_path, safe_storage_component
|
||||
from govoplan_files.backend.storage.provenance import source_provenance_from_metadata, source_revision_from_metadata
|
||||
|
||||
@@ -37,6 +38,7 @@ class ManagedAttachmentFile:
|
||||
checksum_sha256: str
|
||||
size_bytes: int
|
||||
content_type: str | None
|
||||
linked_to_campaign: bool = True
|
||||
source_provenance: dict[str, Any] | None = None
|
||||
source_revision: str | None = None
|
||||
|
||||
@@ -56,6 +58,7 @@ class PreparedCampaignSnapshot:
|
||||
raw_json: dict[str, Any]
|
||||
managed_files_by_local_path: dict[str, ManagedAttachmentFile]
|
||||
shared_assets: list[FileAsset]
|
||||
candidate_assets: list[FileAsset]
|
||||
|
||||
|
||||
def parse_managed_source(value: object) -> tuple[str, str] | None:
|
||||
@@ -117,6 +120,99 @@ def _iter_rule_dicts(attachments: dict[str, Any], raw_json: dict[str, Any]):
|
||||
yield rule
|
||||
|
||||
|
||||
def _managed_base_sources(raw_json: dict[str, Any]) -> list[tuple[str, str, str]]:
|
||||
attachments = raw_json.get("attachments")
|
||||
if not isinstance(attachments, dict):
|
||||
return []
|
||||
base_paths = attachments.get("base_paths")
|
||||
if not isinstance(base_paths, list):
|
||||
return []
|
||||
sources: list[tuple[str, str, str]] = []
|
||||
seen: set[tuple[str, str, str]] = set()
|
||||
for item in base_paths:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
parsed_source = parse_managed_source(item.get("source"))
|
||||
if parsed_source is None:
|
||||
continue
|
||||
owner_type, owner_id = parsed_source
|
||||
old_path = str(item.get("path") or ".").strip() or "."
|
||||
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
|
||||
key = (owner_type, owner_id, logical_root)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
sources.append(key)
|
||||
return sources
|
||||
|
||||
|
||||
def _campaign_linked_asset_ids(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
campaign_id: str,
|
||||
asset_ids: list[str],
|
||||
) -> set[str]:
|
||||
if not asset_ids:
|
||||
return set()
|
||||
linked: set[str] = set()
|
||||
for offset in range(0, len(asset_ids), 900):
|
||||
chunk = asset_ids[offset : offset + 900]
|
||||
rows = (
|
||||
session.query(FileShare.file_asset_id)
|
||||
.filter(
|
||||
FileShare.tenant_id == tenant_id,
|
||||
FileShare.file_asset_id.in_(chunk),
|
||||
FileShare.target_type == "campaign",
|
||||
FileShare.target_id == campaign_id,
|
||||
FileShare.revoked_at.is_(None),
|
||||
)
|
||||
.all()
|
||||
)
|
||||
linked.update(row[0] for row in rows)
|
||||
return linked
|
||||
|
||||
|
||||
def _candidate_assets_for_managed_sources(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
campaign_id: str,
|
||||
raw_json: dict[str, Any],
|
||||
user_id: str,
|
||||
is_admin: bool,
|
||||
shared_assets: list[FileAsset],
|
||||
) -> tuple[list[FileAsset], set[str]]:
|
||||
assets_by_id: dict[str, FileAsset] = {asset.id: asset for asset in shared_assets}
|
||||
for owner_type, owner_id, logical_root in _managed_base_sources(raw_json):
|
||||
ensure_owner_access(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
for asset in list_assets_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
owner_type=owner_type,
|
||||
owner_id=owner_id,
|
||||
path_prefix=logical_root or None,
|
||||
is_admin=is_admin,
|
||||
):
|
||||
assets_by_id.setdefault(asset.id, asset)
|
||||
candidate_assets = sorted(assets_by_id.values(), key=lambda asset: (asset.display_path, asset.updated_at, asset.id))
|
||||
linked_ids = _campaign_linked_asset_ids(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
campaign_id=campaign_id,
|
||||
asset_ids=[asset.id for asset in candidate_assets],
|
||||
)
|
||||
return candidate_assets, linked_ids
|
||||
|
||||
|
||||
def _selected_base_path(
|
||||
rule: dict[str, Any],
|
||||
prepared_by_id: dict[str, tuple[str, str]],
|
||||
@@ -136,6 +232,177 @@ def _selected_base_path(
|
||||
return None
|
||||
|
||||
|
||||
def _prepared_attachment_config(raw_json: dict[str, Any]) -> tuple[dict[str, Any], dict[str, Any], list[Any]]:
|
||||
prepared_json = copy.deepcopy(raw_json if isinstance(raw_json, dict) else {})
|
||||
attachments = prepared_json.get("attachments")
|
||||
if not isinstance(attachments, dict):
|
||||
attachments = {}
|
||||
prepared_json["attachments"] = attachments
|
||||
base_paths = attachments.get("base_paths")
|
||||
if not isinstance(base_paths, list):
|
||||
base_paths = []
|
||||
return prepared_json, attachments, base_paths
|
||||
|
||||
|
||||
def _campaign_snapshot_assets(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
campaign_id: str,
|
||||
prepared_json: dict[str, Any],
|
||||
include_unlinked_candidates: bool,
|
||||
user_id: str,
|
||||
is_admin: bool,
|
||||
) -> tuple[list[FileAsset], list[FileAsset], set[str]]:
|
||||
shared_assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id="",
|
||||
campaign_id=campaign_id,
|
||||
is_admin=True,
|
||||
)
|
||||
if not include_unlinked_candidates:
|
||||
return shared_assets, shared_assets, {asset.id for asset in shared_assets}
|
||||
candidate_assets, linked_asset_ids = _candidate_assets_for_managed_sources(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
campaign_id=campaign_id,
|
||||
raw_json=prepared_json,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
shared_assets=shared_assets,
|
||||
)
|
||||
return shared_assets, candidate_assets, linked_asset_ids
|
||||
|
||||
|
||||
def _assets_grouped_by_owner(assets: list[FileAsset]) -> dict[tuple[str, str], list[FileAsset]]:
|
||||
assets_by_owner: dict[tuple[str, str], list[FileAsset]] = defaultdict(list)
|
||||
for asset in assets:
|
||||
owner_id = _asset_owner_id(asset)
|
||||
if owner_id:
|
||||
assets_by_owner[(asset.owner_type, owner_id)].append(asset)
|
||||
return assets_by_owner
|
||||
|
||||
|
||||
def _materialize_managed_asset(
|
||||
asset: FileAsset,
|
||||
*,
|
||||
owner_id: str,
|
||||
logical_root: str,
|
||||
local_root: Path,
|
||||
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
|
||||
backend: StorageBackend | None,
|
||||
include_bytes: bool,
|
||||
linked_asset_ids: set[str],
|
||||
) -> tuple[str, ManagedAttachmentFile] | None:
|
||||
relative_path = _relative_asset_path(asset, logical_root)
|
||||
if not relative_path:
|
||||
return None
|
||||
target = _safe_local_target(local_root, relative_path)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
version, blob = version_blobs[asset.id]
|
||||
if include_bytes:
|
||||
try:
|
||||
data = backend.get_bytes(blob.storage_key) if backend else b""
|
||||
except StorageBackendError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
target.write_bytes(data)
|
||||
else:
|
||||
target.touch()
|
||||
local_key = str(target.resolve())
|
||||
return local_key, ManagedAttachmentFile(
|
||||
local_path=local_key,
|
||||
asset_id=asset.id,
|
||||
version_id=version.id,
|
||||
blob_id=blob.id,
|
||||
display_path=asset.display_path,
|
||||
relative_path=normalize_logical_path(relative_path),
|
||||
filename=asset.filename,
|
||||
owner_type=asset.owner_type,
|
||||
owner_id=owner_id,
|
||||
checksum_sha256=blob.checksum_sha256,
|
||||
size_bytes=blob.size_bytes,
|
||||
content_type=blob.content_type,
|
||||
linked_to_campaign=asset.id in linked_asset_ids,
|
||||
source_provenance=source_provenance_from_metadata(asset.metadata_),
|
||||
source_revision=source_revision_from_metadata(asset.metadata_),
|
||||
)
|
||||
|
||||
|
||||
def _prepare_managed_base_paths(
|
||||
base_paths: list[Any],
|
||||
*,
|
||||
materialized_root: Path,
|
||||
assets_by_owner: dict[tuple[str, str], list[FileAsset]],
|
||||
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
|
||||
backend: StorageBackend | None,
|
||||
include_bytes: bool,
|
||||
linked_asset_ids: set[str],
|
||||
) -> tuple[
|
||||
dict[str, ManagedAttachmentFile],
|
||||
dict[str, tuple[str, str]],
|
||||
dict[str, list[tuple[str, str]]],
|
||||
tuple[str, str] | None,
|
||||
]:
|
||||
manifest: dict[str, ManagedAttachmentFile] = {}
|
||||
prepared_by_id: dict[str, tuple[str, str]] = {}
|
||||
prepared_by_old_path: dict[str, list[tuple[str, str]]] = {}
|
||||
first_prepared: tuple[str, str] | None = None
|
||||
for index, item in enumerate(base_paths):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
parsed_source = parse_managed_source(item.get("source"))
|
||||
if parsed_source is None:
|
||||
continue
|
||||
owner_type, owner_id = parsed_source
|
||||
old_path = str(item.get("path") or ".").strip() or "."
|
||||
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
|
||||
base_path_id = str(item.get("id") or f"base-path-{index + 1}")
|
||||
local_root = materialized_root / f"{index + 1:03d}-{safe_storage_component(base_path_id)}"
|
||||
local_root.mkdir(parents=True, exist_ok=True)
|
||||
local_root_string = str(local_root.resolve())
|
||||
prepared = (base_path_id, local_root_string)
|
||||
prepared_by_id[base_path_id] = prepared
|
||||
prepared_by_old_path.setdefault(old_path, []).append(prepared)
|
||||
if first_prepared is None:
|
||||
first_prepared = prepared
|
||||
item["path"] = local_root_string
|
||||
for asset in assets_by_owner.get((owner_type, owner_id), []):
|
||||
materialized = _materialize_managed_asset(
|
||||
asset,
|
||||
owner_id=owner_id,
|
||||
logical_root=logical_root,
|
||||
local_root=local_root,
|
||||
version_blobs=version_blobs,
|
||||
backend=backend,
|
||||
include_bytes=include_bytes,
|
||||
linked_asset_ids=linked_asset_ids,
|
||||
)
|
||||
if materialized is not None:
|
||||
local_key, managed_file = materialized
|
||||
manifest[local_key] = managed_file
|
||||
return manifest, prepared_by_id, prepared_by_old_path, first_prepared
|
||||
|
||||
|
||||
def _rewrite_managed_attachment_rules(
|
||||
attachments: dict[str, Any],
|
||||
prepared_json: dict[str, Any],
|
||||
*,
|
||||
prepared_by_id: dict[str, tuple[str, str]],
|
||||
prepared_by_old_path: dict[str, list[tuple[str, str]]],
|
||||
first_prepared: tuple[str, str] | None,
|
||||
) -> None:
|
||||
for rule in _iter_rule_dicts(attachments, prepared_json):
|
||||
selected = _selected_base_path(rule, prepared_by_id, prepared_by_old_path, first_prepared)
|
||||
if selected is None:
|
||||
continue
|
||||
base_path_id, local_root_string = selected
|
||||
rule["base_path_id"] = base_path_id
|
||||
rule["base_dir"] = local_root_string
|
||||
if first_prepared is not None:
|
||||
attachments["base_path"] = first_prepared[1]
|
||||
|
||||
|
||||
def prepare_campaign_snapshot(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -144,6 +411,9 @@ def prepare_campaign_snapshot(
|
||||
raw_json: dict[str, Any],
|
||||
destination: Path,
|
||||
include_bytes: bool,
|
||||
include_unlinked_candidates: bool = False,
|
||||
user_id: str = "",
|
||||
is_admin: bool = False,
|
||||
) -> PreparedCampaignSnapshot:
|
||||
"""Create a temporary file-oriented campaign snapshot for managed attachments.
|
||||
|
||||
@@ -159,101 +429,35 @@ def prepare_campaign_snapshot(
|
||||
materialized_root = destination / "managed-attachments"
|
||||
materialized_root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
prepared_json = copy.deepcopy(raw_json if isinstance(raw_json, dict) else {})
|
||||
attachments = prepared_json.get("attachments")
|
||||
if not isinstance(attachments, dict):
|
||||
attachments = {}
|
||||
prepared_json["attachments"] = attachments
|
||||
base_paths = attachments.get("base_paths")
|
||||
if not isinstance(base_paths, list):
|
||||
base_paths = []
|
||||
|
||||
shared_assets = list_assets_for_user(
|
||||
prepared_json, attachments, base_paths = _prepared_attachment_config(raw_json)
|
||||
shared_assets, candidate_assets, linked_asset_ids = _campaign_snapshot_assets(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id="",
|
||||
campaign_id=campaign_id,
|
||||
is_admin=True,
|
||||
prepared_json=prepared_json,
|
||||
include_unlinked_candidates=include_unlinked_candidates,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
|
||||
assets_by_owner: dict[tuple[str, str], list[FileAsset]] = defaultdict(list)
|
||||
for asset in shared_assets:
|
||||
owner_id = _asset_owner_id(asset)
|
||||
if owner_id:
|
||||
assets_by_owner[(asset.owner_type, owner_id)].append(asset)
|
||||
version_blobs = current_versions_and_blobs(session, shared_assets)
|
||||
assets_by_owner = _assets_grouped_by_owner(candidate_assets)
|
||||
version_blobs = current_versions_and_blobs(session, candidate_assets)
|
||||
backend = get_storage_backend() if include_bytes else None
|
||||
|
||||
manifest: dict[str, ManagedAttachmentFile] = {}
|
||||
prepared_by_id: dict[str, tuple[str, str]] = {}
|
||||
prepared_by_old_path: dict[str, list[tuple[str, str]]] = {}
|
||||
first_prepared: tuple[str, str] | None = None
|
||||
|
||||
for index, item in enumerate(base_paths):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
parsed_source = parse_managed_source(item.get("source"))
|
||||
if parsed_source is None:
|
||||
continue
|
||||
owner_type, owner_id = parsed_source
|
||||
old_path = str(item.get("path") or ".").strip() or "."
|
||||
logical_root = "" if old_path in {"", ".", "/"} else normalize_folder(old_path)
|
||||
base_path_id = str(item.get("id") or f"base-path-{index + 1}")
|
||||
local_root = materialized_root / f"{index + 1:03d}-{safe_storage_component(base_path_id)}"
|
||||
local_root.mkdir(parents=True, exist_ok=True)
|
||||
local_root_string = str(local_root.resolve())
|
||||
|
||||
prepared = (base_path_id, local_root_string)
|
||||
prepared_by_id[base_path_id] = prepared
|
||||
prepared_by_old_path.setdefault(old_path, []).append(prepared)
|
||||
if first_prepared is None:
|
||||
first_prepared = prepared
|
||||
|
||||
item["path"] = local_root_string
|
||||
|
||||
for asset in assets_by_owner.get((owner_type, owner_id), []):
|
||||
relative_path = _relative_asset_path(asset, logical_root)
|
||||
if not relative_path:
|
||||
continue
|
||||
target = _safe_local_target(local_root, relative_path)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
version, blob = version_blobs[asset.id]
|
||||
if include_bytes:
|
||||
try:
|
||||
data = backend.get_bytes(blob.storage_key) if backend else b""
|
||||
except StorageBackendError as exc:
|
||||
raise FileStorageError(str(exc)) from exc
|
||||
target.write_bytes(data)
|
||||
else:
|
||||
target.touch()
|
||||
local_key = str(target.resolve())
|
||||
manifest[local_key] = ManagedAttachmentFile(
|
||||
local_path=local_key,
|
||||
asset_id=asset.id,
|
||||
version_id=version.id,
|
||||
blob_id=blob.id,
|
||||
display_path=asset.display_path,
|
||||
relative_path=normalize_logical_path(relative_path),
|
||||
filename=asset.filename,
|
||||
owner_type=asset.owner_type,
|
||||
owner_id=owner_id,
|
||||
checksum_sha256=blob.checksum_sha256,
|
||||
size_bytes=blob.size_bytes,
|
||||
content_type=blob.content_type,
|
||||
source_provenance=source_provenance_from_metadata(asset.metadata_),
|
||||
source_revision=source_revision_from_metadata(asset.metadata_),
|
||||
manifest, prepared_by_id, prepared_by_old_path, first_prepared = _prepare_managed_base_paths(
|
||||
base_paths,
|
||||
materialized_root=materialized_root,
|
||||
assets_by_owner=assets_by_owner,
|
||||
version_blobs=version_blobs,
|
||||
backend=backend,
|
||||
include_bytes=include_bytes,
|
||||
linked_asset_ids=linked_asset_ids,
|
||||
)
|
||||
_rewrite_managed_attachment_rules(
|
||||
attachments,
|
||||
prepared_json,
|
||||
prepared_by_id=prepared_by_id,
|
||||
prepared_by_old_path=prepared_by_old_path,
|
||||
first_prepared=first_prepared,
|
||||
)
|
||||
|
||||
for rule in _iter_rule_dicts(attachments, prepared_json):
|
||||
selected = _selected_base_path(rule, prepared_by_id, prepared_by_old_path, first_prepared)
|
||||
if selected is None:
|
||||
continue
|
||||
base_path_id, local_root_string = selected
|
||||
rule["base_path_id"] = base_path_id
|
||||
rule["base_dir"] = local_root_string
|
||||
|
||||
if first_prepared is not None:
|
||||
attachments["base_path"] = first_prepared[1]
|
||||
|
||||
snapshot_path = destination / "campaign.json"
|
||||
snapshot_path.write_text(json.dumps(prepared_json, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
@@ -262,6 +466,7 @@ def prepare_campaign_snapshot(
|
||||
raw_json=prepared_json,
|
||||
managed_files_by_local_path=manifest,
|
||||
shared_assets=shared_assets,
|
||||
candidate_assets=candidate_assets,
|
||||
)
|
||||
|
||||
|
||||
@@ -273,7 +478,10 @@ def prepared_campaign_snapshot(
|
||||
campaign_id: str,
|
||||
raw_json: dict[str, Any],
|
||||
include_bytes: bool,
|
||||
prefix: str = "multimailer-managed-campaign-",
|
||||
prefix: str = "govoplan-managed-campaign-",
|
||||
include_unlinked_candidates: bool = False,
|
||||
user_id: str = "",
|
||||
is_admin: bool = False,
|
||||
) -> Iterator[PreparedCampaignSnapshot]:
|
||||
temp_dir = Path(tempfile.mkdtemp(prefix=prefix))
|
||||
try:
|
||||
@@ -284,6 +492,9 @@ def prepared_campaign_snapshot(
|
||||
raw_json=raw_json,
|
||||
destination=temp_dir,
|
||||
include_bytes=include_bytes,
|
||||
include_unlinked_candidates=include_unlinked_candidates,
|
||||
user_id=user_id,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
finally:
|
||||
shutil.rmtree(temp_dir, ignore_errors=True)
|
||||
@@ -301,6 +512,53 @@ def managed_match_payloads(
|
||||
return payloads
|
||||
|
||||
|
||||
def share_assets_with_campaign(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
campaign_id: str,
|
||||
file_ids: list[str],
|
||||
user_id: str,
|
||||
is_admin: bool = False,
|
||||
) -> list[dict[str, Any]]:
|
||||
unique_ids = list(dict.fromkeys(file_id for file_id in file_ids if file_id))
|
||||
if not unique_ids:
|
||||
return []
|
||||
assets = [
|
||||
get_asset_for_user(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
user_id=user_id,
|
||||
asset_id=file_id,
|
||||
require_write=True,
|
||||
is_admin=is_admin,
|
||||
)
|
||||
for file_id in unique_ids
|
||||
]
|
||||
shares = share_files(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
assets=assets,
|
||||
target_type="campaign",
|
||||
target_id=campaign_id,
|
||||
permission="read",
|
||||
user_id=user_id,
|
||||
)
|
||||
session.flush()
|
||||
return [
|
||||
{
|
||||
"id": share.id,
|
||||
"file_asset_id": share.file_asset_id,
|
||||
"target_type": share.target_type,
|
||||
"target_id": share.target_id,
|
||||
"permission": share.permission,
|
||||
"created_at": share.created_at.isoformat() if share.created_at else None,
|
||||
"revoked_at": share.revoked_at.isoformat() if share.revoked_at else None,
|
||||
}
|
||||
for share in shares
|
||||
]
|
||||
|
||||
|
||||
def public_attachment_summary_payload(value: Any) -> dict[str, Any]:
|
||||
"""Return an attachment summary without temporary materialization paths.
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import PurePosixPath
|
||||
from typing import Any, Iterable
|
||||
|
||||
@@ -24,6 +25,15 @@ def _candidate_match_keys(raw_match: str) -> set[str]:
|
||||
AttachmentUseKey = tuple[str, str, str, str]
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _AttachmentBatchRefs:
|
||||
managed_by_job: dict[str, list[dict[str, object]]] = field(default_factory=lambda: defaultdict(list))
|
||||
fallback_attachments_by_job: dict[str, list[dict[str, object]]] = field(default_factory=lambda: defaultdict(list))
|
||||
asset_ids: set[str] = field(default_factory=set)
|
||||
version_ids: set[str] = field(default_factory=set)
|
||||
blob_ids: set[str] = field(default_factory=set)
|
||||
|
||||
|
||||
def _attachment_use_key(*, job_id: str, file_version_id: str, filename_used: str, stage: str) -> AttachmentUseKey:
|
||||
return job_id, file_version_id, filename_used, stage
|
||||
|
||||
@@ -134,22 +144,48 @@ def record_campaign_attachment_uses_for_jobs(
|
||||
if not job_list:
|
||||
return
|
||||
|
||||
managed_by_job: dict[str, list[dict[str, object]]] = defaultdict(list)
|
||||
fallback_attachments_by_job: dict[str, list[dict[str, object]]] = defaultdict(list)
|
||||
refs = _collect_attachment_batch_refs(job_list)
|
||||
job_by_id = {job.id: job for job in job_list}
|
||||
asset_ids: set[str] = set()
|
||||
version_ids: set[str] = set()
|
||||
blob_ids: set[str] = set()
|
||||
known_keys = _known_use_keys_for_jobs(session, job_list, stage=stage)
|
||||
assets_by_id, versions_by_id, blobs_by_id = _load_managed_attachment_entities(session, refs)
|
||||
|
||||
for job in job_list:
|
||||
_record_managed_attachment_uses(
|
||||
session,
|
||||
job_by_id=job_by_id,
|
||||
managed_by_job=refs.managed_by_job,
|
||||
assets_by_id=assets_by_id,
|
||||
versions_by_id=versions_by_id,
|
||||
blobs_by_id=blobs_by_id,
|
||||
stage=stage,
|
||||
known_keys=known_keys,
|
||||
)
|
||||
_record_fallback_attachment_uses(
|
||||
session,
|
||||
job_by_id=job_by_id,
|
||||
fallback_attachments_by_job=refs.fallback_attachments_by_job,
|
||||
stage=stage,
|
||||
known_keys=known_keys,
|
||||
)
|
||||
|
||||
|
||||
def _collect_attachment_batch_refs(jobs: list[CampaignJobLike]) -> _AttachmentBatchRefs:
|
||||
refs = _AttachmentBatchRefs()
|
||||
for job in jobs:
|
||||
attachments = job.resolved_attachments or []
|
||||
if not isinstance(attachments, list):
|
||||
continue
|
||||
for attachment in attachments:
|
||||
if not isinstance(attachment, dict):
|
||||
continue
|
||||
if not _collect_managed_attachment_refs(refs, job.id, attachment):
|
||||
refs.fallback_attachments_by_job[job.id].append(attachment)
|
||||
return refs
|
||||
|
||||
|
||||
def _collect_managed_attachment_refs(refs: _AttachmentBatchRefs, job_id: str, attachment: dict[str, object]) -> bool:
|
||||
managed_matches = attachment.get("managed_matches")
|
||||
if isinstance(managed_matches, list) and managed_matches:
|
||||
if not isinstance(managed_matches, list) or not managed_matches:
|
||||
return False
|
||||
for item in managed_matches:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
@@ -158,38 +194,41 @@ def record_campaign_attachment_uses_for_jobs(
|
||||
blob_id = str(item.get("blob_id") or "")
|
||||
if not asset_id or not version_id or not blob_id:
|
||||
continue
|
||||
managed_by_job[job.id].append(item)
|
||||
asset_ids.add(asset_id)
|
||||
version_ids.add(version_id)
|
||||
blob_ids.add(blob_id)
|
||||
else:
|
||||
fallback_attachments_by_job[job.id].append(attachment)
|
||||
refs.managed_by_job[job_id].append(item)
|
||||
refs.asset_ids.add(asset_id)
|
||||
refs.version_ids.add(version_id)
|
||||
refs.blob_ids.add(blob_id)
|
||||
return True
|
||||
|
||||
assets_by_id = {
|
||||
item.id: item
|
||||
for item in session.query(FileAsset).filter(FileAsset.id.in_(asset_ids)).all()
|
||||
} if asset_ids else {}
|
||||
versions_by_id = {
|
||||
item.id: item
|
||||
for item in session.query(FileVersion).filter(FileVersion.id.in_(version_ids)).all()
|
||||
} if version_ids else {}
|
||||
blobs_by_id = {
|
||||
item.id: item
|
||||
for item in session.query(FileBlob).filter(FileBlob.id.in_(blob_ids)).all()
|
||||
} if blob_ids else {}
|
||||
known_keys = _known_use_keys_for_jobs(session, job_list, stage=stage)
|
||||
|
||||
def _load_managed_attachment_entities(
|
||||
session: Session,
|
||||
refs: _AttachmentBatchRefs,
|
||||
) -> tuple[dict[str, FileAsset], dict[str, FileVersion], dict[str, FileBlob]]:
|
||||
assets_by_id = {item.id: item for item in session.query(FileAsset).filter(FileAsset.id.in_(refs.asset_ids)).all()} if refs.asset_ids else {}
|
||||
versions_by_id = {item.id: item for item in session.query(FileVersion).filter(FileVersion.id.in_(refs.version_ids)).all()} if refs.version_ids else {}
|
||||
blobs_by_id = {item.id: item for item in session.query(FileBlob).filter(FileBlob.id.in_(refs.blob_ids)).all()} if refs.blob_ids else {}
|
||||
return assets_by_id, versions_by_id, blobs_by_id
|
||||
|
||||
|
||||
def _record_managed_attachment_uses(
|
||||
session: Session,
|
||||
*,
|
||||
job_by_id: dict[str, CampaignJobLike],
|
||||
managed_by_job: dict[str, list[dict[str, object]]],
|
||||
assets_by_id: dict[str, FileAsset],
|
||||
versions_by_id: dict[str, FileVersion],
|
||||
blobs_by_id: dict[str, FileBlob],
|
||||
stage: str,
|
||||
known_keys: set[AttachmentUseKey],
|
||||
) -> None:
|
||||
for job_id, items in managed_by_job.items():
|
||||
job = job_by_id[job_id]
|
||||
for item in items:
|
||||
asset = assets_by_id.get(str(item.get("asset_id") or ""))
|
||||
version = versions_by_id.get(str(item.get("version_id") or ""))
|
||||
blob = blobs_by_id.get(str(item.get("blob_id") or ""))
|
||||
if not asset or not version or not blob:
|
||||
continue
|
||||
if asset.tenant_id != job.tenant_id or version.tenant_id != job.tenant_id or blob.tenant_id != job.tenant_id:
|
||||
continue
|
||||
if version.file_asset_id != asset.id or version.blob_id != blob.id:
|
||||
if not _managed_attachment_entities_match_job(job, asset, version, blob):
|
||||
continue
|
||||
_add_use(
|
||||
session,
|
||||
@@ -202,29 +241,79 @@ def record_campaign_attachment_uses_for_jobs(
|
||||
known_keys=known_keys,
|
||||
)
|
||||
|
||||
|
||||
def _managed_attachment_entities_match_job(
|
||||
job: CampaignJobLike,
|
||||
asset: FileAsset | None,
|
||||
version: FileVersion | None,
|
||||
blob: FileBlob | None,
|
||||
) -> bool:
|
||||
if not asset or not version or not blob:
|
||||
return False
|
||||
if asset.tenant_id != job.tenant_id or version.tenant_id != job.tenant_id or blob.tenant_id != job.tenant_id:
|
||||
return False
|
||||
return version.file_asset_id == asset.id and version.blob_id == blob.id
|
||||
|
||||
|
||||
def _record_fallback_attachment_uses(
|
||||
session: Session,
|
||||
*,
|
||||
job_by_id: dict[str, CampaignJobLike],
|
||||
fallback_attachments_by_job: dict[str, list[dict[str, object]]],
|
||||
stage: str,
|
||||
known_keys: set[AttachmentUseKey],
|
||||
) -> None:
|
||||
assets_by_campaign: dict[tuple[str, str], dict[str, FileAsset]] = {}
|
||||
version_blobs_by_campaign: dict[tuple[str, str], dict[str, tuple[FileVersion, FileBlob]]] = {}
|
||||
for job_id, attachments in fallback_attachments_by_job.items():
|
||||
job = job_by_id[job_id]
|
||||
campaign_key = (job.tenant_id, job.campaign_id)
|
||||
by_key, version_blobs = _fallback_campaign_assets(
|
||||
session,
|
||||
job,
|
||||
campaign_key=campaign_key,
|
||||
assets_by_campaign=assets_by_campaign,
|
||||
version_blobs_by_campaign=version_blobs_by_campaign,
|
||||
)
|
||||
for attachment in attachments:
|
||||
_record_fallback_attachment(session, job, attachment, by_key=by_key, version_blobs=version_blobs, stage=stage, known_keys=known_keys)
|
||||
|
||||
|
||||
def _fallback_campaign_assets(
|
||||
session: Session,
|
||||
job: CampaignJobLike,
|
||||
*,
|
||||
campaign_key: tuple[str, str],
|
||||
assets_by_campaign: dict[tuple[str, str], dict[str, FileAsset]],
|
||||
version_blobs_by_campaign: dict[tuple[str, str], dict[str, tuple[FileVersion, FileBlob]]],
|
||||
) -> tuple[dict[str, FileAsset], dict[str, tuple[FileVersion, FileBlob]]]:
|
||||
by_key = assets_by_campaign.get(campaign_key)
|
||||
if by_key is None:
|
||||
assets = list_assets_for_user(
|
||||
session,
|
||||
tenant_id=job.tenant_id,
|
||||
user_id="",
|
||||
campaign_id=job.campaign_id,
|
||||
is_admin=True,
|
||||
)
|
||||
by_key = {}
|
||||
assets = list_assets_for_user(session, tenant_id=job.tenant_id, user_id="", campaign_id=job.campaign_id, is_admin=True)
|
||||
by_key = _asset_lookup_by_path_and_filename(assets)
|
||||
assets_by_campaign[campaign_key] = by_key
|
||||
version_blobs_by_campaign[campaign_key] = current_versions_and_blobs(session, assets)
|
||||
return by_key, version_blobs_by_campaign[campaign_key]
|
||||
|
||||
|
||||
def _asset_lookup_by_path_and_filename(assets: list[FileAsset]) -> dict[str, FileAsset]:
|
||||
by_key: dict[str, FileAsset] = {}
|
||||
for asset in assets:
|
||||
by_key[asset.display_path.strip("/")] = asset
|
||||
by_key.setdefault(asset.filename, asset)
|
||||
assets_by_campaign[campaign_key] = by_key
|
||||
version_blobs_by_campaign[campaign_key] = current_versions_and_blobs(session, assets)
|
||||
version_blobs = version_blobs_by_campaign[campaign_key]
|
||||
return by_key
|
||||
|
||||
for attachment in attachments:
|
||||
|
||||
def _record_fallback_attachment(
|
||||
session: Session,
|
||||
job: CampaignJobLike,
|
||||
attachment: dict[str, object],
|
||||
*,
|
||||
by_key: dict[str, FileAsset],
|
||||
version_blobs: dict[str, tuple[FileVersion, FileBlob]],
|
||||
stage: str,
|
||||
known_keys: set[AttachmentUseKey],
|
||||
) -> None:
|
||||
matches = attachment.get("matches") if isinstance(attachment.get("matches"), list) else []
|
||||
for raw in matches:
|
||||
if not isinstance(raw, str):
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
@@ -9,11 +9,23 @@ from importlib import import_module
|
||||
import mimetypes
|
||||
from typing import Any
|
||||
from urllib.parse import quote, unquote, urljoin, urlsplit
|
||||
import xml.etree.ElementTree as ET
|
||||
import xml.etree.ElementTree as ET # nosec B405 - typing/element creation only; parsing uses defusedxml below.
|
||||
|
||||
from defusedxml import ElementTree as SafeElementTree
|
||||
import httpx
|
||||
|
||||
from govoplan_core.security.outbound_http import (
|
||||
OutboundHttpError,
|
||||
validate_unpinned_sdk_host,
|
||||
validate_unpinned_sdk_http_url,
|
||||
)
|
||||
|
||||
from govoplan_files.backend.storage.http_client import ConnectorHttpError, request_connector_bytes
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
ConnectorDeploymentConfigurationError,
|
||||
connector_ca_bundle_path,
|
||||
connector_secret_env_value,
|
||||
validate_connector_tls_metadata,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
|
||||
|
||||
@@ -53,7 +65,7 @@ class ConnectorBrowseItem:
|
||||
}
|
||||
|
||||
|
||||
def browse_connector_profile(profile: ConnectorProfile, *, path: str | None = None, library_id: str | None = None) -> list[ConnectorBrowseItem]:
|
||||
def browse_connector_profile(profile: ConnectorProfile, *, path: str | None = None, library_id: str | None = None, continuation_token: str | None = None) -> list[ConnectorBrowseItem]:
|
||||
browse_path = normalize_connector_browse_path(path)
|
||||
static_items = _static_listing(profile, path=browse_path, library_id=library_id)
|
||||
if static_items is not None:
|
||||
@@ -66,6 +78,8 @@ def browse_connector_profile(profile: ConnectorProfile, *, path: str | None = No
|
||||
return _browse_webdav(profile, path=browse_path)
|
||||
if profile.provider == "smb":
|
||||
return _browse_smb(profile, path=browse_path)
|
||||
if profile.provider == "s3":
|
||||
return _browse_s3(profile, path=browse_path, library_id=library_id, continuation_token=continuation_token)
|
||||
raise ConnectorBrowseUnsupported(f"Read-only browsing is not implemented for {profile.provider} connector profiles yet")
|
||||
|
||||
|
||||
@@ -168,14 +182,22 @@ def _browse_webdav(profile: ConnectorProfile, *, path: str) -> list[ConnectorBro
|
||||
</prop>
|
||||
</propfind>"""
|
||||
try:
|
||||
response = httpx.request("PROPFIND", url, headers=headers, content=body, auth=auth, timeout=15.0)
|
||||
except httpx.HTTPError as exc:
|
||||
response = request_connector_bytes(
|
||||
"PROPFIND",
|
||||
url,
|
||||
headers=headers,
|
||||
content=body,
|
||||
auth=auth,
|
||||
timeout=15.0,
|
||||
label="WebDAV browse",
|
||||
)
|
||||
except ConnectorHttpError as exc:
|
||||
raise ConnectorBrowseError(f"Connector browse failed: {exc}") from exc
|
||||
if response.status_code in {401, 403}:
|
||||
raise ConnectorBrowseError("Connector credentials were rejected")
|
||||
if response.status_code not in {200, 207}:
|
||||
raise ConnectorBrowseError(f"Connector browse failed with HTTP {response.status_code}")
|
||||
return parse_webdav_multistatus(root_url=root_url, current_path=path, payload=response.text)
|
||||
return parse_webdav_multistatus(root_url=root_url, current_path=path, payload=response.content)
|
||||
|
||||
|
||||
def _browse_smb(profile: ConnectorProfile, *, path: str) -> list[ConnectorBrowseItem]:
|
||||
@@ -222,6 +244,245 @@ def _browse_smb(profile: ConnectorProfile, *, path: str) -> list[ConnectorBrowse
|
||||
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
|
||||
|
||||
|
||||
def _browse_s3(profile: ConnectorProfile, *, path: str, library_id: str | None, continuation_token: str | None) -> list[ConnectorBrowseItem]:
|
||||
client = _s3_client(profile)
|
||||
bucket = _s3_bucket(profile, library_id)
|
||||
if not bucket:
|
||||
try:
|
||||
payload = client.list_buckets()
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorBrowseError(f"S3 connector browse failed: {exc}") from exc
|
||||
buckets = payload.get("Buckets") if isinstance(payload, Mapping) else None
|
||||
if not isinstance(buckets, list):
|
||||
raise ConnectorBrowseError("S3 connector returned invalid bucket list")
|
||||
return sorted(
|
||||
(
|
||||
ConnectorBrowseItem(
|
||||
kind="library",
|
||||
name=_clean(item.get("Name")) or "",
|
||||
path=_clean(item.get("Name")) or "",
|
||||
external_id=_clean(item.get("Name")),
|
||||
modified_at=_timestamp(item.get("CreationDate")),
|
||||
metadata={"bucket": _clean(item.get("Name"))},
|
||||
)
|
||||
for item in buckets
|
||||
if isinstance(item, Mapping) and _clean(item.get("Name"))
|
||||
),
|
||||
key=lambda item: item.name.casefold(),
|
||||
)
|
||||
prefix = _s3_object_key(profile, path, directory=True)
|
||||
params: dict[str, object] = {
|
||||
"Bucket": bucket,
|
||||
"Prefix": prefix,
|
||||
"Delimiter": "/",
|
||||
"MaxKeys": _s3_max_keys(profile),
|
||||
}
|
||||
next_page_request = _clean(continuation_token) or _metadata_string(profile, "continuation_token")
|
||||
if next_page_request:
|
||||
params["ContinuationToken"] = next_page_request
|
||||
try:
|
||||
payload = client.list_objects_v2(**params)
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorBrowseError(f"S3 connector browse failed: {exc}") from exc
|
||||
if not isinstance(payload, Mapping):
|
||||
raise ConnectorBrowseError("S3 connector returned invalid object listing")
|
||||
items = [
|
||||
*_s3_prefix_items(bucket=bucket, browse_path=path, base_prefix=prefix, prefixes=payload.get("CommonPrefixes")),
|
||||
*_s3_object_items(bucket=bucket, browse_path=path, base_prefix=prefix, objects=payload.get("Contents")),
|
||||
]
|
||||
next_token = _clean(payload.get("NextContinuationToken"))
|
||||
if next_token and items:
|
||||
last = items[-1]
|
||||
items[-1] = ConnectorBrowseItem(
|
||||
kind=last.kind,
|
||||
name=last.name,
|
||||
path=last.path,
|
||||
external_id=last.external_id,
|
||||
external_url=last.external_url,
|
||||
size_bytes=last.size_bytes,
|
||||
content_type=last.content_type,
|
||||
modified_at=last.modified_at,
|
||||
etag=last.etag,
|
||||
metadata={**dict(last.metadata), "next_continuation_token": next_token, "listing_truncated": True},
|
||||
)
|
||||
return sorted(items, key=lambda item: (item.kind != "folder", item.name.casefold(), item.path.casefold()))
|
||||
|
||||
|
||||
def _s3_client(profile: ConnectorProfile) -> Any:
|
||||
if profile.secret_ref:
|
||||
raise ConnectorBrowseError("Secret-ref S3 credentials need a runtime secret resolver before live browsing")
|
||||
if profile.endpoint_url:
|
||||
try:
|
||||
endpoint_url = validate_unpinned_sdk_http_url(
|
||||
profile.endpoint_url,
|
||||
label="S3 connector endpoint",
|
||||
)
|
||||
except OutboundHttpError as exc:
|
||||
raise ConnectorBrowseError(str(exc)) from exc
|
||||
else:
|
||||
raise ConnectorBrowseError(
|
||||
"S3 connector endpoint discovery uses an SDK transport that cannot guarantee connection-time DNS/IP "
|
||||
"pinning; live S3 access is disabled until that transport supports pinning"
|
||||
)
|
||||
try:
|
||||
boto3 = import_module("boto3")
|
||||
config_module = import_module("botocore.config")
|
||||
except ImportError as exc:
|
||||
raise ConnectorBrowseUnsupported("S3 connector browsing requires the optional boto3 dependency") from exc
|
||||
kwargs: dict[str, object] = {"endpoint_url": endpoint_url}
|
||||
region = _metadata_string(profile, "region") or _metadata_string(profile, "aws_region")
|
||||
if region:
|
||||
kwargs["region_name"] = region
|
||||
access_key = profile.username or _metadata_env(profile, "access_key_id_env") or _metadata_string(profile, "access_key_id")
|
||||
secret_key = _profile_password(profile) or _metadata_env(profile, "secret_access_key_env")
|
||||
session_token = _profile_token(profile) or _metadata_env(profile, "session_token_env")
|
||||
if access_key:
|
||||
kwargs["aws_access_key_id"] = access_key
|
||||
if secret_key:
|
||||
kwargs["aws_secret_access_key"] = secret_key
|
||||
if session_token:
|
||||
kwargs["aws_session_token"] = session_token
|
||||
verify = _s3_verify(profile)
|
||||
if verify is not None:
|
||||
kwargs["verify"] = verify
|
||||
addressing_style = _s3_addressing_style(profile)
|
||||
if addressing_style:
|
||||
kwargs["config"] = config_module.Config(s3={"addressing_style": addressing_style})
|
||||
try:
|
||||
return boto3.client("s3", **kwargs)
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorBrowseError(f"S3 connector could not be initialized: {exc}") from exc
|
||||
|
||||
|
||||
def _s3_bucket(profile: ConnectorProfile, library_id: str | None = None) -> str | None:
|
||||
return _metadata_string(profile, "bucket") or _metadata_string(profile, "bucket_name") or _clean(library_id)
|
||||
|
||||
|
||||
def _s3_object_key(profile: ConnectorProfile, path: str, *, directory: bool = False) -> str:
|
||||
base_prefix = normalize_connector_browse_path(profile.base_path or _metadata_string(profile, "base_prefix"))
|
||||
browse_path = normalize_connector_browse_path(path)
|
||||
key = "/".join(part for part in (base_prefix, browse_path) if part)
|
||||
if directory and key:
|
||||
return key.rstrip("/") + "/"
|
||||
return key
|
||||
|
||||
|
||||
def _s3_prefix_items(
|
||||
*,
|
||||
bucket: str,
|
||||
browse_path: str,
|
||||
base_prefix: str,
|
||||
prefixes: object,
|
||||
) -> list[ConnectorBrowseItem]:
|
||||
if not isinstance(prefixes, list):
|
||||
return []
|
||||
items: list[ConnectorBrowseItem] = []
|
||||
for item in prefixes:
|
||||
if not isinstance(item, Mapping):
|
||||
continue
|
||||
key = _clean(item.get("Prefix"))
|
||||
if not key:
|
||||
continue
|
||||
relative = _s3_relative_key(key, base_prefix=base_prefix)
|
||||
name = _path_name(relative)
|
||||
if not name:
|
||||
continue
|
||||
path = _join_browse_path(browse_path, name)
|
||||
items.append(
|
||||
ConnectorBrowseItem(
|
||||
kind="folder",
|
||||
name=name,
|
||||
path=path,
|
||||
external_id=f"{bucket}:{key}",
|
||||
metadata={"bucket": bucket, "key": key},
|
||||
)
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def _s3_object_items(
|
||||
*,
|
||||
bucket: str,
|
||||
browse_path: str,
|
||||
base_prefix: str,
|
||||
objects: object,
|
||||
) -> list[ConnectorBrowseItem]:
|
||||
if not isinstance(objects, list):
|
||||
return []
|
||||
items: list[ConnectorBrowseItem] = []
|
||||
for item in objects:
|
||||
if not isinstance(item, Mapping):
|
||||
continue
|
||||
key = _clean(item.get("Key"))
|
||||
if not key or key == base_prefix:
|
||||
continue
|
||||
relative = _s3_relative_key(key, base_prefix=base_prefix)
|
||||
name = _path_name(relative)
|
||||
if not name:
|
||||
continue
|
||||
path = _join_browse_path(browse_path, name)
|
||||
items.append(
|
||||
ConnectorBrowseItem(
|
||||
kind="file",
|
||||
name=name,
|
||||
path=path,
|
||||
external_id=f"{bucket}:{key}",
|
||||
size_bytes=_int(item.get("Size")),
|
||||
content_type=mimetypes.guess_type(name)[0],
|
||||
modified_at=_timestamp(item.get("LastModified")),
|
||||
etag=_clean(item.get("ETag")),
|
||||
metadata={
|
||||
"bucket": bucket,
|
||||
"key": key,
|
||||
**({"storage_class": item["StorageClass"]} if "StorageClass" in item else {}),
|
||||
},
|
||||
)
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def _s3_relative_key(key: str, *, base_prefix: str) -> str:
|
||||
clean_key = key.rstrip("/")
|
||||
clean_base = base_prefix.rstrip("/")
|
||||
if clean_base and clean_key.startswith(f"{clean_base}/"):
|
||||
return clean_key[len(clean_base) + 1 :]
|
||||
return clean_key
|
||||
|
||||
|
||||
def _s3_max_keys(profile: ConnectorProfile) -> int:
|
||||
configured = _int(profile.metadata.get("max_keys"))
|
||||
if configured is None:
|
||||
return 1000
|
||||
return max(1, min(configured, 1000))
|
||||
|
||||
|
||||
def _s3_verify(profile: ConnectorProfile) -> bool | str | None:
|
||||
try:
|
||||
validate_connector_tls_metadata(profile.metadata)
|
||||
except ConnectorDeploymentConfigurationError as exc:
|
||||
raise ConnectorBrowseError(str(exc)) from exc
|
||||
ca_bundle = _metadata_string(profile, "ca_bundle")
|
||||
if ca_bundle:
|
||||
try:
|
||||
return connector_ca_bundle_path(ca_bundle)
|
||||
except ConnectorDeploymentConfigurationError as exc:
|
||||
raise ConnectorBrowseError(str(exc)) from exc
|
||||
if "verify_tls" in profile.metadata:
|
||||
return _metadata_bool(profile, "verify_tls", default=True)
|
||||
if "tls_verify" in profile.metadata:
|
||||
return _metadata_bool(profile, "tls_verify", default=True)
|
||||
return None
|
||||
|
||||
|
||||
def _s3_addressing_style(profile: ConnectorProfile) -> str | None:
|
||||
style = _metadata_string(profile, "addressing_style")
|
||||
if style in {"path", "virtual", "auto"}:
|
||||
return style
|
||||
if _metadata_bool(profile, "path_style", default=False):
|
||||
return "path"
|
||||
return None
|
||||
|
||||
|
||||
def _seafile_headers(profile: ConnectorProfile) -> dict[str, str]:
|
||||
token = _seafile_token(profile)
|
||||
return {"Authorization": f"Token {token}", "Accept": "application/json"}
|
||||
@@ -261,16 +522,24 @@ def _request_json(
|
||||
data: Mapping[str, str] | None = None,
|
||||
) -> object:
|
||||
try:
|
||||
response = httpx.request(method, url, headers=dict(headers or {}), params=params, data=data, timeout=15.0)
|
||||
except httpx.HTTPError as exc:
|
||||
response = request_connector_bytes(
|
||||
method,
|
||||
url,
|
||||
headers=dict(headers or {}),
|
||||
params=params,
|
||||
data=data,
|
||||
timeout=15.0,
|
||||
label="Seafile API",
|
||||
)
|
||||
except ConnectorHttpError as exc:
|
||||
raise ConnectorBrowseError(f"Connector browse failed: {exc}") from exc
|
||||
if response.status_code in {401, 403}:
|
||||
raise ConnectorBrowseError("Connector credentials were rejected")
|
||||
if response.status_code not in {200, 201}:
|
||||
raise ConnectorBrowseError(f"Connector browse failed with HTTP {response.status_code}")
|
||||
try:
|
||||
return response.json()
|
||||
except ValueError as exc:
|
||||
return json.loads(response.content)
|
||||
except (UnicodeDecodeError, ValueError) as exc:
|
||||
raise ConnectorBrowseError("Connector returned invalid JSON") from exc
|
||||
|
||||
|
||||
@@ -444,11 +713,18 @@ def _metadata_bool(profile: ConnectorProfile, key: str, *, default: bool = False
|
||||
return str(value).strip().casefold() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _env_required(name: str, profile_id: str) -> str:
|
||||
value = _clean(os.environ.get(name))
|
||||
if not value:
|
||||
raise ConnectorBrowseError(f"Connector profile {profile_id} is missing required credential environment")
|
||||
return value
|
||||
def _metadata_env(profile: ConnectorProfile, key: str) -> str | None:
|
||||
env_name = _metadata_string(profile, key)
|
||||
if not env_name:
|
||||
return None
|
||||
return _env_required(env_name, profile)
|
||||
|
||||
|
||||
def _env_required(name: str, profile: ConnectorProfile) -> str:
|
||||
try:
|
||||
return connector_secret_env_value(name, source_kind=profile.source_kind)
|
||||
except ConnectorDeploymentConfigurationError as exc:
|
||||
raise ConnectorBrowseError(f"Connector profile {profile.id}: {exc}") from exc
|
||||
|
||||
|
||||
def normalize_connector_browse_path(value: object) -> str:
|
||||
@@ -505,6 +781,11 @@ def _smb_location(profile: ConnectorProfile) -> _SmbLocation:
|
||||
server = _clean(parsed.hostname)
|
||||
if not server:
|
||||
raise ConnectorBrowseError("SMB connector endpoint_url must include a server")
|
||||
port = parsed.port or _int(profile.metadata.get("port")) or 445
|
||||
try:
|
||||
validate_unpinned_sdk_host(server, port=port, label="SMB connector endpoint")
|
||||
except OutboundHttpError as exc:
|
||||
raise ConnectorBrowseError(str(exc)) from exc
|
||||
path_parts = [part for part in unquote(parsed.path or "").strip("/").split("/") if part]
|
||||
if not path_parts:
|
||||
raise ConnectorBrowseError("SMB connector endpoint_url must include a share name")
|
||||
@@ -514,7 +795,7 @@ def _smb_location(profile: ConnectorProfile) -> _SmbLocation:
|
||||
return _SmbLocation(
|
||||
server=server,
|
||||
share=path_parts[0],
|
||||
port=parsed.port or _int(profile.metadata.get("port")) or 445,
|
||||
port=port,
|
||||
root_path=normalize_connector_browse_path("/".join(root_parts)),
|
||||
)
|
||||
|
||||
@@ -550,7 +831,7 @@ def _profile_password(profile: ConnectorProfile) -> str | None:
|
||||
if profile.password_value:
|
||||
return profile.password_value
|
||||
if profile.password_env:
|
||||
return _env_required(profile.password_env, profile.id)
|
||||
return _env_required(profile.password_env, profile)
|
||||
return None
|
||||
|
||||
|
||||
@@ -558,7 +839,7 @@ def _profile_token(profile: ConnectorProfile) -> str | None:
|
||||
if profile.token_value:
|
||||
return profile.token_value
|
||||
if profile.token_env:
|
||||
return _env_required(profile.token_env, profile.id)
|
||||
return _env_required(profile.token_env, profile)
|
||||
return None
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,337 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from sqlalchemy import inspect
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.audit.logging import audit_event
|
||||
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorCredentialDeletionResult:
|
||||
changed: bool
|
||||
affected_profiles: tuple[FileConnectorProfile, ...] = ()
|
||||
|
||||
|
||||
def delete_connector_credential_row(
|
||||
session: Session,
|
||||
row: FileConnectorCredential,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
user_id: str | None = None,
|
||||
api_key_id: str | None = None,
|
||||
) -> ConnectorCredentialDeletionResult:
|
||||
"""Scrub a credential tombstone and disable every profile that used it.
|
||||
|
||||
``secret_ref`` predates a Files-owned secret-provider contract. It may be
|
||||
shared or deployment-owned, so it is detached locally and explicitly
|
||||
audited without ever being passed to a provider delete operation.
|
||||
"""
|
||||
|
||||
dependent_profiles = (
|
||||
tuple(
|
||||
session.query(FileConnectorProfile)
|
||||
.filter(FileConnectorProfile.credential_profile_id == row.id)
|
||||
.order_by(FileConnectorProfile.id.asc())
|
||||
.all()
|
||||
)
|
||||
if inspect(session.get_bind()).has_table(FileConnectorProfile.__tablename__)
|
||||
else ()
|
||||
)
|
||||
affected_profiles: list[FileConnectorProfile] = []
|
||||
for profile in dependent_profiles:
|
||||
if _delete_connector_profile_row(
|
||||
session,
|
||||
profile,
|
||||
deletion_reason="credential_deleted",
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
):
|
||||
affected_profiles.append(profile)
|
||||
|
||||
deleted_secret_kinds = _encrypted_secret_kinds(row)
|
||||
removed_reference_kinds = _credential_reference_kinds(row)
|
||||
removed_metadata = bool(row.metadata_)
|
||||
changed = _credential_row_requires_deletion(row)
|
||||
if changed:
|
||||
_scrub_credential_row(row, user_id=user_id)
|
||||
session.add(row)
|
||||
_audit_credential_deletion(
|
||||
session,
|
||||
row,
|
||||
deletion_reason=deletion_reason,
|
||||
deleted_secret_kinds=deleted_secret_kinds,
|
||||
removed_reference_kinds=removed_reference_kinds,
|
||||
removed_metadata=removed_metadata,
|
||||
affected_profile_count=len(affected_profiles),
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
)
|
||||
session.flush()
|
||||
return ConnectorCredentialDeletionResult(
|
||||
changed=changed or bool(affected_profiles),
|
||||
affected_profiles=tuple(affected_profiles),
|
||||
)
|
||||
|
||||
|
||||
def delete_connector_profile_row(
|
||||
session: Session,
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
user_id: str | None = None,
|
||||
api_key_id: str | None = None,
|
||||
) -> bool:
|
||||
changed = _delete_connector_profile_row(
|
||||
session,
|
||||
row,
|
||||
deletion_reason=deletion_reason,
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
)
|
||||
session.flush()
|
||||
return changed
|
||||
|
||||
|
||||
def delete_connector_credentials_for_retirement(session: Session) -> int:
|
||||
"""Scrub and audit stored connector material before destructive retirement.
|
||||
|
||||
Legacy external references are detached and audited as non-owned. Files
|
||||
never sends those arbitrary references to a provider delete operation.
|
||||
"""
|
||||
|
||||
inspector = inspect(session.get_bind())
|
||||
profiles = (
|
||||
session.query(FileConnectorProfile).order_by(FileConnectorProfile.id.asc()).all()
|
||||
if inspector.has_table(FileConnectorProfile.__tablename__)
|
||||
else []
|
||||
)
|
||||
credentials = (
|
||||
session.query(FileConnectorCredential).order_by(FileConnectorCredential.id.asc()).all()
|
||||
if inspector.has_table(FileConnectorCredential.__tablename__)
|
||||
else []
|
||||
)
|
||||
deleted = 0
|
||||
for profile in profiles:
|
||||
if not _profile_row_has_credential_material(profile):
|
||||
continue
|
||||
if _delete_connector_profile_row(
|
||||
session,
|
||||
profile,
|
||||
deletion_reason="module_data_retired",
|
||||
):
|
||||
deleted += 1
|
||||
for credential in credentials:
|
||||
if not _credential_row_has_material(credential):
|
||||
continue
|
||||
result = delete_connector_credential_row(
|
||||
session,
|
||||
credential,
|
||||
deletion_reason="module_data_retired",
|
||||
)
|
||||
if result.changed:
|
||||
deleted += 1
|
||||
session.flush()
|
||||
return deleted
|
||||
|
||||
|
||||
def _delete_connector_profile_row(
|
||||
session: Session,
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
user_id: str | None = None,
|
||||
api_key_id: str | None = None,
|
||||
) -> bool:
|
||||
deleted_secret_kinds = _encrypted_secret_kinds(row)
|
||||
removed_reference_kinds = _profile_reference_kinds(row)
|
||||
removed_metadata = bool(row.metadata_)
|
||||
changed = _profile_row_requires_deletion(row)
|
||||
if not changed:
|
||||
return False
|
||||
_scrub_profile_row(row, user_id=user_id)
|
||||
session.add(row)
|
||||
_audit_profile_deletion(
|
||||
session,
|
||||
row,
|
||||
deletion_reason=deletion_reason,
|
||||
deleted_secret_kinds=deleted_secret_kinds,
|
||||
removed_reference_kinds=removed_reference_kinds,
|
||||
removed_metadata=removed_metadata,
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
def _scrub_credential_row(row: FileConnectorCredential, *, user_id: str | None) -> None:
|
||||
row.enabled = False
|
||||
row.credential_mode = "none"
|
||||
row.username = None
|
||||
row.password_encrypted = None
|
||||
row.token_encrypted = None
|
||||
row.password_env = None
|
||||
row.token_env = None
|
||||
row.secret_ref = None
|
||||
row.metadata_ = {}
|
||||
row.updated_by_user_id = user_id
|
||||
|
||||
|
||||
def _scrub_profile_row(row: FileConnectorProfile, *, user_id: str | None) -> None:
|
||||
row.enabled = False
|
||||
row.credential_profile_id = None
|
||||
row.credential_mode = "none"
|
||||
row.username = None
|
||||
row.password_encrypted = None
|
||||
row.token_encrypted = None
|
||||
row.password_env = None
|
||||
row.token_env = None
|
||||
row.secret_ref = None
|
||||
row.metadata_ = {}
|
||||
row.updated_by_user_id = user_id
|
||||
|
||||
|
||||
def _credential_row_requires_deletion(row: FileConnectorCredential) -> bool:
|
||||
return bool(row.enabled or _credential_row_has_material(row))
|
||||
|
||||
|
||||
def _profile_row_requires_deletion(row: FileConnectorProfile) -> bool:
|
||||
return bool(row.enabled or _profile_row_has_credential_material(row))
|
||||
|
||||
|
||||
def _credential_row_has_material(row: FileConnectorCredential) -> bool:
|
||||
return bool(
|
||||
row.username
|
||||
or row.password_encrypted
|
||||
or row.token_encrypted
|
||||
or row.password_env
|
||||
or row.token_env
|
||||
or row.secret_ref
|
||||
or row.metadata_
|
||||
or row.credential_mode not in {"", "none", "anonymous"}
|
||||
)
|
||||
|
||||
|
||||
def _profile_row_has_credential_material(row: FileConnectorProfile) -> bool:
|
||||
return bool(
|
||||
row.credential_profile_id
|
||||
or row.username
|
||||
or row.password_encrypted
|
||||
or row.token_encrypted
|
||||
or row.password_env
|
||||
or row.token_env
|
||||
or row.secret_ref
|
||||
or row.metadata_
|
||||
or row.credential_mode not in {"", "none", "anonymous"}
|
||||
)
|
||||
|
||||
|
||||
def _encrypted_secret_kinds(row: FileConnectorCredential | FileConnectorProfile) -> list[str]:
|
||||
kinds: list[str] = []
|
||||
if row.password_encrypted:
|
||||
kinds.append("password")
|
||||
if row.token_encrypted:
|
||||
kinds.append("token")
|
||||
return kinds
|
||||
|
||||
|
||||
def _credential_reference_kinds(row: FileConnectorCredential | FileConnectorProfile) -> list[str]:
|
||||
kinds: list[str] = []
|
||||
if row.password_env:
|
||||
kinds.append("password_env")
|
||||
if row.token_env:
|
||||
kinds.append("token_env")
|
||||
if row.secret_ref:
|
||||
kinds.append("unowned_external_secret_ref")
|
||||
return kinds
|
||||
|
||||
|
||||
def _profile_reference_kinds(row: FileConnectorProfile) -> list[str]:
|
||||
kinds = _credential_reference_kinds(row)
|
||||
if row.credential_profile_id:
|
||||
kinds.append("credential_profile")
|
||||
return kinds
|
||||
|
||||
|
||||
def _storage_backend(deleted_secret_kinds: list[str], removed_reference_kinds: list[str]) -> str:
|
||||
if "unowned_external_secret_ref" in removed_reference_kinds:
|
||||
return "unowned_external_reference_detached"
|
||||
if deleted_secret_kinds:
|
||||
return "encrypted_database"
|
||||
if removed_reference_kinds:
|
||||
return "reference_only"
|
||||
return "none"
|
||||
|
||||
|
||||
def _audit_credential_deletion(
|
||||
session: Session,
|
||||
row: FileConnectorCredential,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
deleted_secret_kinds: list[str],
|
||||
removed_reference_kinds: list[str],
|
||||
removed_metadata: bool,
|
||||
affected_profile_count: int,
|
||||
user_id: str | None,
|
||||
api_key_id: str | None,
|
||||
) -> None:
|
||||
audit_event(
|
||||
session,
|
||||
tenant_id=row.tenant_id,
|
||||
scope=_audit_scope(row.scope_type),
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
action="files.connector_credential_deleted",
|
||||
object_type="file_connector_credential",
|
||||
object_id=row.id,
|
||||
details={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
"storage_backend": _storage_backend(deleted_secret_kinds, removed_reference_kinds),
|
||||
"deleted_secret_kinds": deleted_secret_kinds,
|
||||
"removed_reference_kinds": removed_reference_kinds,
|
||||
"removed_metadata": removed_metadata,
|
||||
"affected_profile_count": affected_profile_count,
|
||||
"deletion_reason": deletion_reason,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _audit_profile_deletion(
|
||||
session: Session,
|
||||
row: FileConnectorProfile,
|
||||
*,
|
||||
deletion_reason: str,
|
||||
deleted_secret_kinds: list[str],
|
||||
removed_reference_kinds: list[str],
|
||||
removed_metadata: bool,
|
||||
user_id: str | None,
|
||||
api_key_id: str | None,
|
||||
) -> None:
|
||||
audit_event(
|
||||
session,
|
||||
tenant_id=row.tenant_id,
|
||||
scope=_audit_scope(row.scope_type),
|
||||
user_id=user_id,
|
||||
api_key_id=api_key_id,
|
||||
action="files.connector_profile_deleted",
|
||||
object_type="file_connector_profile",
|
||||
object_id=row.id,
|
||||
details={
|
||||
"scope_type": row.scope_type,
|
||||
"scope_id": row.scope_id,
|
||||
"provider": row.provider,
|
||||
"storage_backend": _storage_backend(deleted_secret_kinds, removed_reference_kinds),
|
||||
"deleted_secret_kinds": deleted_secret_kinds,
|
||||
"removed_reference_kinds": removed_reference_kinds,
|
||||
"removed_metadata": removed_metadata,
|
||||
"deletion_reason": deletion_reason,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _audit_scope(scope_type: str) -> str:
|
||||
return "system" if scope_type == "system" else "tenant"
|
||||
@@ -10,6 +10,7 @@ from govoplan_core.core.policy import normalize_policy_scope_type, policy_source
|
||||
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
|
||||
from govoplan_files.backend.db.models import FileConnectorCredential
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_deployment import reject_api_controlled_deployment_references
|
||||
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload
|
||||
from govoplan_files.backend.storage.connector_profiles import supported_connector_providers
|
||||
|
||||
@@ -51,9 +52,13 @@ class ConnectorCredential:
|
||||
|
||||
@property
|
||||
def credentials_configured(self) -> bool:
|
||||
if not self.enabled:
|
||||
return False
|
||||
if self.credential_mode.casefold() in {"", "none", "anonymous"}:
|
||||
return True
|
||||
return bool(self.secret_ref or self.password_value or self.token_value or self.password_env or self.token_env)
|
||||
# Environment references are deliberately unavailable to API-managed
|
||||
# credential rows. Legacy rows remain visible but fail closed.
|
||||
return bool(self.secret_ref or self.password_value or self.token_value)
|
||||
|
||||
def to_response(self) -> dict[str, Any]:
|
||||
return {
|
||||
@@ -182,6 +187,12 @@ def create_connector_credential_row(
|
||||
policy: Mapping[str, Any] | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
) -> FileConnectorCredential:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
metadata=metadata,
|
||||
)
|
||||
clean_id = _normalize_id(credential_id)
|
||||
if session.get(FileConnectorCredential, clean_id) is not None:
|
||||
raise FileStorageError(f"Connector credential already exists: {clean_id}")
|
||||
@@ -231,6 +242,18 @@ def update_connector_credential_row(
|
||||
clear_password: bool = False,
|
||||
clear_token: bool = False,
|
||||
) -> FileConnectorCredential:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
metadata=metadata,
|
||||
)
|
||||
if secret_ref is not None and _clean(secret_ref) != _clean(row.secret_ref):
|
||||
if _clean(row.secret_ref):
|
||||
raise FileStorageError(
|
||||
"An existing external secret reference cannot be replaced or cleared until Files can prove "
|
||||
"provider ownership and confirm provider-side deletion"
|
||||
)
|
||||
if label is not None:
|
||||
row.label = _normalize_label(label)
|
||||
if provider is not None:
|
||||
@@ -265,14 +288,6 @@ def update_connector_credential_row(
|
||||
return row
|
||||
|
||||
|
||||
def deactivate_connector_credential_row(session: Session, row: FileConnectorCredential, *, user_id: str | None) -> FileConnectorCredential:
|
||||
row.enabled = False
|
||||
row.updated_by_user_id = user_id
|
||||
session.add(row)
|
||||
session.flush()
|
||||
return row
|
||||
|
||||
|
||||
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:
|
||||
clean_scope_type = normalize_policy_scope_type(scope_type)
|
||||
clean_scope_id = _clean(scope_id)
|
||||
|
||||
241
src/govoplan_files/backend/storage/connector_deployment.py
Normal file
241
src/govoplan_files/backend/storage/connector_deployment.py
Normal file
@@ -0,0 +1,241 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
_SECRET_ENV_ALLOWLIST = "GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST" # noqa: S105 # nosec B105 - configuration key.
|
||||
_CA_BUNDLE_ALLOWLIST = "GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST"
|
||||
_ENV_NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
|
||||
_SECRET_ENV_METADATA_KEYS = frozenset(
|
||||
{
|
||||
"access_key_id_env",
|
||||
"secret_access_key_env",
|
||||
"session_token_env",
|
||||
}
|
||||
)
|
||||
_SECRET_VALUE_METADATA_KEYS = frozenset(
|
||||
{
|
||||
"password",
|
||||
"token",
|
||||
"access_token",
|
||||
"auth_token",
|
||||
"bearer_token",
|
||||
"refresh_token",
|
||||
"api_key",
|
||||
"access_key",
|
||||
"access_key_id",
|
||||
"secret_key",
|
||||
"secret_access_key",
|
||||
"session_token",
|
||||
}
|
||||
)
|
||||
_DEVELOPMENT_ENVIRONMENTS = frozenset({"dev", "development", "local", "test", "testing"})
|
||||
|
||||
|
||||
class ConnectorDeploymentConfigurationError(ValueError):
|
||||
"""Raised when connector data crosses a deployment-owned trust boundary."""
|
||||
|
||||
|
||||
def connector_secret_env_value(name: str, *, source_kind: str) -> str:
|
||||
clean_name = _validate_secret_env_reference(name, source_kind=source_kind)
|
||||
value = os.environ.get(clean_name)
|
||||
if value is None or value == "":
|
||||
raise ConnectorDeploymentConfigurationError("Connector credential environment variable is not configured")
|
||||
return value
|
||||
|
||||
|
||||
def connector_secret_env_available(name: str | None, *, source_kind: str) -> bool:
|
||||
if not name:
|
||||
return False
|
||||
try:
|
||||
clean_name = _validate_secret_env_reference(name, source_kind=source_kind)
|
||||
except ConnectorDeploymentConfigurationError:
|
||||
return False
|
||||
return bool(os.environ.get(clean_name))
|
||||
|
||||
|
||||
def validate_deployment_connector_references(
|
||||
*,
|
||||
source_kind: str,
|
||||
password_env: str | None = None,
|
||||
token_env: str | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
) -> None:
|
||||
"""Validate references in deployment-owned connector configuration."""
|
||||
|
||||
for name in (password_env, token_env):
|
||||
if name:
|
||||
_validate_secret_env_reference(name, source_kind=source_kind)
|
||||
for key in _SECRET_ENV_METADATA_KEYS:
|
||||
name = _clean((metadata or {}).get(key))
|
||||
if name:
|
||||
_validate_secret_env_reference(name, source_kind=source_kind)
|
||||
validate_connector_tls_metadata(metadata)
|
||||
|
||||
|
||||
def reject_api_controlled_deployment_references(
|
||||
*,
|
||||
password_env: str | None = None,
|
||||
token_env: str | None = None,
|
||||
secret_ref: str | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
) -> None:
|
||||
"""Reject process-secret selectors controlled through tenant-facing APIs."""
|
||||
|
||||
if _clean(password_env) or _clean(token_env):
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
|
||||
)
|
||||
if _clean(secret_ref):
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"External secret references cannot be managed through the Files API until Files can prove "
|
||||
"provider ownership and confirm provider-side deletion"
|
||||
)
|
||||
for key, value in _metadata_entries(metadata or {}):
|
||||
clean_key = str(key).strip().casefold()
|
||||
if _clean(value) and (clean_key in _SECRET_ENV_METADATA_KEYS or clean_key.endswith("_env")):
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Environment-backed credentials may only be declared in deployment-owned connector configuration"
|
||||
)
|
||||
if _clean(value) and (
|
||||
clean_key in _SECRET_VALUE_METADATA_KEYS
|
||||
or clean_key.endswith(("_password", "_secret", "_api_key"))
|
||||
):
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Connector credential values must use the dedicated encrypted credential fields, not metadata"
|
||||
)
|
||||
validate_connector_tls_metadata(metadata)
|
||||
|
||||
|
||||
def _metadata_entries(value: object) -> list[tuple[str, object]]:
|
||||
entries: list[tuple[str, object]] = []
|
||||
if isinstance(value, Mapping):
|
||||
for key, item in value.items():
|
||||
entries.append((str(key), item))
|
||||
entries.extend(_metadata_entries(item))
|
||||
elif isinstance(value, (list, tuple)):
|
||||
for item in value:
|
||||
entries.extend(_metadata_entries(item))
|
||||
return entries
|
||||
|
||||
|
||||
def connector_ca_bundle_path(value: str) -> str:
|
||||
clean_value = _clean(value)
|
||||
if not clean_value:
|
||||
raise ConnectorDeploymentConfigurationError("Connector CA bundle path is empty")
|
||||
candidate = Path(clean_value)
|
||||
if not candidate.is_absolute():
|
||||
raise ConnectorDeploymentConfigurationError("Connector CA bundle paths must be absolute")
|
||||
try:
|
||||
resolved = candidate.resolve(strict=True)
|
||||
except OSError as exc:
|
||||
raise ConnectorDeploymentConfigurationError("Connector CA bundle path does not exist") from exc
|
||||
allowed = _allowed_ca_bundle_paths()
|
||||
if resolved not in allowed:
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
f"Connector CA bundle path is not listed in {_CA_BUNDLE_ALLOWLIST}"
|
||||
)
|
||||
if not resolved.is_file():
|
||||
raise ConnectorDeploymentConfigurationError("Connector CA bundle path must be a regular file")
|
||||
return str(resolved)
|
||||
|
||||
|
||||
def validate_connector_tls_metadata(metadata: Mapping[str, Any] | None) -> None:
|
||||
values = metadata or {}
|
||||
ca_bundle = _clean(values.get("ca_bundle"))
|
||||
if ca_bundle:
|
||||
connector_ca_bundle_path(ca_bundle)
|
||||
for key in ("verify_tls", "tls_verify"):
|
||||
if key in values and not _as_bool(values.get(key), default=True) and not _development_runtime():
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Connector TLS certificate verification may only be disabled in dev/test environments"
|
||||
)
|
||||
|
||||
|
||||
def connector_effective_endpoint_url(
|
||||
*,
|
||||
provider: str | None,
|
||||
endpoint_url: str | None,
|
||||
metadata: Mapping[str, Any] | None,
|
||||
) -> str | None:
|
||||
"""Return the endpoint that connector I/O will actually use."""
|
||||
|
||||
clean_provider = (provider or "").strip().casefold()
|
||||
values = metadata or {}
|
||||
webdav_url = _clean(values.get("webdav_endpoint_url"))
|
||||
browse_protocol = (_clean(values.get("browse_protocol")) or "").casefold()
|
||||
if webdav_url and (clean_provider in {"seafile", "webdav", "nextcloud"} or browse_protocol == "webdav"):
|
||||
return webdav_url
|
||||
return _clean(endpoint_url)
|
||||
|
||||
|
||||
def _validate_secret_env_reference(name: str, *, source_kind: str) -> str:
|
||||
if source_kind.strip().casefold() != "settings":
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
"Environment-backed credentials may only be used by deployment-owned connector configuration"
|
||||
)
|
||||
clean_name = name.strip()
|
||||
if not _ENV_NAME.fullmatch(clean_name):
|
||||
raise ConnectorDeploymentConfigurationError("Connector credential environment variable name is invalid")
|
||||
if clean_name not in _allowed_secret_env_names():
|
||||
raise ConnectorDeploymentConfigurationError(
|
||||
f"Connector credential environment variable is not listed in {_SECRET_ENV_ALLOWLIST}"
|
||||
)
|
||||
return clean_name
|
||||
|
||||
|
||||
def _allowed_secret_env_names() -> frozenset[str]:
|
||||
raw = os.environ.get(_SECRET_ENV_ALLOWLIST, "")
|
||||
names = frozenset(item.strip() for item in raw.split(",") if item.strip())
|
||||
invalid = sorted(name for name in names if not _ENV_NAME.fullmatch(name))
|
||||
if invalid:
|
||||
raise ConnectorDeploymentConfigurationError(f"{_SECRET_ENV_ALLOWLIST} contains an invalid variable name")
|
||||
return names
|
||||
|
||||
|
||||
def _allowed_ca_bundle_paths() -> frozenset[Path]:
|
||||
raw = os.environ.get(_CA_BUNDLE_ALLOWLIST, "")
|
||||
paths: set[Path] = set()
|
||||
for item in (part.strip() for part in raw.split(",")):
|
||||
if not item:
|
||||
continue
|
||||
path = Path(item)
|
||||
if not path.is_absolute():
|
||||
raise ConnectorDeploymentConfigurationError(f"{_CA_BUNDLE_ALLOWLIST} requires absolute paths")
|
||||
try:
|
||||
paths.add(path.resolve(strict=True))
|
||||
except OSError as exc:
|
||||
raise ConnectorDeploymentConfigurationError(f"{_CA_BUNDLE_ALLOWLIST} contains a missing path") from exc
|
||||
return frozenset(paths)
|
||||
|
||||
|
||||
def _development_runtime() -> bool:
|
||||
app_env = os.environ.get("APP_ENV", "dev").strip().casefold()
|
||||
install_profile = os.environ.get("GOVOPLAN_INSTALL_PROFILE", "").strip().casefold()
|
||||
return app_env in _DEVELOPMENT_ENVIRONMENTS and (
|
||||
not install_profile or install_profile in _DEVELOPMENT_ENVIRONMENTS
|
||||
)
|
||||
|
||||
|
||||
def _as_bool(value: object, *, default: bool) -> bool:
|
||||
if value is None:
|
||||
return default
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
clean = str(value).strip().casefold()
|
||||
if clean in {"1", "true", "yes", "on"}:
|
||||
return True
|
||||
if clean in {"0", "false", "no", "off"}:
|
||||
return False
|
||||
raise ConnectorDeploymentConfigurationError("Connector TLS verification setting must be true or false")
|
||||
|
||||
|
||||
def _clean(value: object) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
clean = str(value).strip()
|
||||
return clean or None
|
||||
@@ -4,7 +4,7 @@ from dataclasses import dataclass, field
|
||||
import mimetypes
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from govoplan_core.security.outbound_http import response_limit
|
||||
|
||||
from govoplan_files.backend.storage.connector_browse import (
|
||||
ConnectorBrowseError,
|
||||
@@ -21,12 +21,16 @@ from govoplan_files.backend.storage.connector_browse import (
|
||||
_smb_stat_size,
|
||||
_smb_unc_path,
|
||||
_smbclient_module,
|
||||
_s3_bucket,
|
||||
_s3_client,
|
||||
_s3_object_key,
|
||||
_seafile_headers,
|
||||
_seafile_url,
|
||||
_webdav_url,
|
||||
normalize_connector_browse_path,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
from govoplan_files.backend.storage.http_client import ConnectorHttpError, request_connector_bytes
|
||||
from govoplan_files.backend.storage.paths import filename_from_path
|
||||
|
||||
|
||||
@@ -56,6 +60,7 @@ def read_connector_file(
|
||||
path: str,
|
||||
max_bytes: int,
|
||||
) -> ConnectorDownloadedFile:
|
||||
max_bytes = min(max_bytes, response_limit("file"))
|
||||
if profile.provider == "seafile":
|
||||
if _metadata_string(profile, "browse_protocol") == "webdav" or _metadata_string(profile, "webdav_endpoint_url"):
|
||||
return _read_webdav_file(profile, path=path, max_bytes=max_bytes)
|
||||
@@ -64,6 +69,8 @@ def read_connector_file(
|
||||
return _read_webdav_file(profile, path=path, max_bytes=max_bytes)
|
||||
if profile.provider == "smb":
|
||||
return _read_smb_file(profile, path=path, max_bytes=max_bytes)
|
||||
if profile.provider == "s3":
|
||||
return _read_s3_file(profile, library_id=library_id, path=path, max_bytes=max_bytes)
|
||||
raise ConnectorImportUnsupported(f"Connector file import is not implemented for {profile.provider} profiles yet")
|
||||
|
||||
|
||||
@@ -97,8 +104,15 @@ def _read_seafile_file(profile: ConnectorProfile, *, library_id: str, path: str,
|
||||
if not isinstance(download_url, str) or not download_url.strip():
|
||||
raise ConnectorImportError("Seafile did not return a file download URL")
|
||||
try:
|
||||
response = httpx.request("GET", download_url, timeout=30.0)
|
||||
except httpx.HTTPError as exc:
|
||||
response = request_connector_bytes(
|
||||
"GET",
|
||||
download_url,
|
||||
timeout=30.0,
|
||||
kind="file",
|
||||
max_bytes=max_bytes,
|
||||
label="Seafile file download",
|
||||
)
|
||||
except ConnectorHttpError as exc:
|
||||
raise ConnectorImportError(f"Seafile file download failed: {exc}") from exc
|
||||
if response.status_code != 200:
|
||||
raise ConnectorImportError(f"Seafile file download failed with HTTP {response.status_code}")
|
||||
@@ -144,8 +158,17 @@ def _read_webdav_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -
|
||||
elif profile.credential_mode.casefold() not in {"", "none", "anonymous"} and profile.secret_ref:
|
||||
raise ConnectorImportError("Secret-ref WebDAV credentials need a runtime secret resolver before live import")
|
||||
try:
|
||||
response = httpx.request("GET", url, headers=headers, auth=auth, timeout=30.0)
|
||||
except httpx.HTTPError as exc:
|
||||
response = request_connector_bytes(
|
||||
"GET",
|
||||
url,
|
||||
headers=headers,
|
||||
auth=auth,
|
||||
timeout=30.0,
|
||||
kind="file",
|
||||
max_bytes=max_bytes,
|
||||
label="WebDAV file download",
|
||||
)
|
||||
except ConnectorHttpError as exc:
|
||||
raise ConnectorImportError(f"WebDAV file download failed: {exc}") from exc
|
||||
if response.status_code in {401, 403}:
|
||||
raise ConnectorImportError("Connector credentials were rejected")
|
||||
@@ -171,12 +194,12 @@ def _read_webdav_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -
|
||||
|
||||
|
||||
def _read_smb_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -> ConnectorDownloadedFile:
|
||||
location = _smb_location(profile)
|
||||
file_path = normalize_connector_browse_path(path)
|
||||
if not file_path:
|
||||
raise ConnectorImportError("SMB import requires a file path")
|
||||
unc_path = _smb_unc_path(location, file_path)
|
||||
try:
|
||||
location = _smb_location(profile)
|
||||
unc_path = _smb_unc_path(location, file_path)
|
||||
smbclient = _smbclient_module()
|
||||
kwargs = _smb_client_kwargs(profile, location)
|
||||
stat_result = smbclient.stat(unc_path, **kwargs)
|
||||
@@ -213,6 +236,114 @@ def _read_smb_file(profile: ConnectorProfile, *, path: str, max_bytes: int) -> C
|
||||
)
|
||||
|
||||
|
||||
def _s3_import_client(profile: ConnectorProfile) -> Any:
|
||||
try:
|
||||
return _s3_client(profile)
|
||||
except ConnectorBrowseUnsupported as exc:
|
||||
raise ConnectorImportUnsupported(str(exc)) from exc
|
||||
except ConnectorBrowseError as exc:
|
||||
raise ConnectorImportError(str(exc)) from exc
|
||||
|
||||
|
||||
def _s3_object_detail(client: Any, *, bucket: str, key: str, max_bytes: int) -> dict[str, Any]:
|
||||
try:
|
||||
detail = client.head_object(Bucket=bucket, Key=key)
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorImportError(f"S3 object metadata lookup failed: {exc}") from exc
|
||||
if not isinstance(detail, dict):
|
||||
raise ConnectorImportError("S3 connector returned invalid object metadata")
|
||||
size = _int(detail.get("ContentLength"))
|
||||
if size is not None and size > max_bytes:
|
||||
raise ConnectorImportError(f"S3 object exceeds limit of {max_bytes} bytes")
|
||||
return detail
|
||||
|
||||
|
||||
def _download_s3_object(
|
||||
client: Any,
|
||||
*,
|
||||
bucket: str,
|
||||
key: str,
|
||||
version_id: str | None,
|
||||
max_bytes: int,
|
||||
) -> tuple[Any, bytes]:
|
||||
request: dict[str, object] = {"Bucket": bucket, "Key": key}
|
||||
if version_id:
|
||||
request["VersionId"] = version_id
|
||||
try:
|
||||
response = client.get_object(**request)
|
||||
body = response.get("Body")
|
||||
data = body.read(max_bytes + 1) if hasattr(body, "read") else bytes(response.get("Body") or b"")
|
||||
except Exception as exc: # pragma: no cover - concrete exception types are dependency-version specific
|
||||
raise ConnectorImportError(f"S3 object download failed: {exc}") from exc
|
||||
if len(data) > max_bytes:
|
||||
raise ConnectorImportError(f"S3 object exceeds limit of {max_bytes} bytes")
|
||||
return response, data
|
||||
|
||||
|
||||
def _s3_download_metadata(
|
||||
detail: dict[str, Any],
|
||||
*,
|
||||
bucket: str,
|
||||
key: str,
|
||||
version_id: str | None,
|
||||
etag: str | None,
|
||||
size: int,
|
||||
) -> dict[str, Any]:
|
||||
metadata: dict[str, Any] = {
|
||||
"bucket": bucket,
|
||||
"key": key,
|
||||
"version_id": version_id,
|
||||
"etag": etag,
|
||||
"size": size,
|
||||
}
|
||||
for source_key, target_key in (
|
||||
("ChecksumSHA256", "checksum_sha256"),
|
||||
("ChecksumCRC32", "checksum_crc32"),
|
||||
("StorageClass", "storage_class"),
|
||||
):
|
||||
if source_key in detail:
|
||||
metadata[target_key] = detail[source_key]
|
||||
return metadata
|
||||
|
||||
|
||||
def _read_s3_file(profile: ConnectorProfile, *, library_id: str, path: str, max_bytes: int) -> ConnectorDownloadedFile:
|
||||
bucket = _s3_bucket(profile, library_id)
|
||||
if not bucket:
|
||||
raise ConnectorImportError("S3 import requires a bucket in library_id or profile metadata")
|
||||
key = _s3_object_key(profile, path)
|
||||
if not key:
|
||||
raise ConnectorImportError("S3 import requires an object key")
|
||||
client = _s3_import_client(profile)
|
||||
detail = _s3_object_detail(client, bucket=bucket, key=key, max_bytes=max_bytes)
|
||||
version_id = _clean(detail.get("VersionId"))
|
||||
response, data = _download_s3_object(
|
||||
client,
|
||||
bucket=bucket,
|
||||
key=key,
|
||||
version_id=version_id,
|
||||
max_bytes=max_bytes,
|
||||
)
|
||||
content_type = _clean(response.get("ContentType") if isinstance(response, dict) else None) or _clean(detail.get("ContentType")) or mimetypes.guess_type(key)[0]
|
||||
etag = _clean(response.get("ETag") if isinstance(response, dict) else None) or _clean(detail.get("ETag"))
|
||||
filename = filename_from_path(key)
|
||||
return ConnectorDownloadedFile(
|
||||
filename=filename,
|
||||
data=data,
|
||||
content_type=content_type,
|
||||
revision=version_id or etag or _clean(detail.get("LastModified")),
|
||||
external_id=f"{bucket}:{key}",
|
||||
external_url=f"s3://{bucket}/{key}",
|
||||
metadata=_s3_download_metadata(
|
||||
detail,
|
||||
bucket=bucket,
|
||||
key=key,
|
||||
version_id=version_id,
|
||||
etag=etag,
|
||||
size=len(data),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _int(value: object) -> int | None:
|
||||
if value is None or value == "":
|
||||
return None
|
||||
|
||||
@@ -9,6 +9,7 @@ from govoplan_core.core.policy import normalize_policy_scope_type
|
||||
from govoplan_core.security.secrets import decrypt_secret, encrypt_secret
|
||||
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
|
||||
from govoplan_files.backend.storage.common import FileStorageError
|
||||
from govoplan_files.backend.storage.connector_deployment import reject_api_controlled_deployment_references
|
||||
from govoplan_files.backend.storage.connector_credential_store import connector_credential_from_row, credential_rows_by_id
|
||||
from govoplan_files.backend.storage.connector_policy import connector_policy_sources_from_payload
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile, supported_connector_providers
|
||||
@@ -124,6 +125,12 @@ def create_connector_profile_row(
|
||||
policy: Mapping[str, Any] | None = None,
|
||||
metadata: Mapping[str, Any] | None = None,
|
||||
) -> FileConnectorProfile:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
metadata=metadata,
|
||||
)
|
||||
clean_id = _normalize_profile_id(profile_id)
|
||||
if session.get(FileConnectorProfile, clean_id) is not None:
|
||||
raise FileStorageError(f"Connector profile already exists: {clean_id}")
|
||||
@@ -181,6 +188,18 @@ def update_connector_profile_row(
|
||||
clear_password: bool = False,
|
||||
clear_token: bool = False,
|
||||
) -> FileConnectorProfile:
|
||||
reject_api_controlled_deployment_references(
|
||||
password_env=password_env,
|
||||
token_env=token_env,
|
||||
secret_ref=secret_ref,
|
||||
metadata=metadata,
|
||||
)
|
||||
if secret_ref is not None and _clean(secret_ref) != _clean(row.secret_ref):
|
||||
if _clean(row.secret_ref):
|
||||
raise FileStorageError(
|
||||
"An existing external secret reference cannot be replaced or cleared until Files can prove "
|
||||
"provider ownership and confirm provider-side deletion"
|
||||
)
|
||||
if label is not None:
|
||||
row.label = _normalize_label(label)
|
||||
if provider is not None:
|
||||
@@ -223,14 +242,6 @@ def update_connector_profile_row(
|
||||
return row
|
||||
|
||||
|
||||
def deactivate_connector_profile_row(session: Session, row: FileConnectorProfile, *, user_id: str | None) -> FileConnectorProfile:
|
||||
row.enabled = False
|
||||
row.updated_by_user_id = user_id
|
||||
session.add(row)
|
||||
session.flush()
|
||||
return row
|
||||
|
||||
|
||||
def _normalize_scope(*, tenant_id: str, scope_type: str, scope_id: str | None) -> tuple[str, str | None, str | None]:
|
||||
clean_scope_type = normalize_policy_scope_type(scope_type)
|
||||
clean_scope_id = _clean(scope_id)
|
||||
|
||||
@@ -7,13 +7,26 @@ from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from govoplan_core.core.policy import normalize_policy_scope_type, policy_source_path
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
connector_secret_env_available,
|
||||
validate_deployment_connector_references,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_policy import ConnectorPolicySource, connector_policy_sources_from_payload
|
||||
|
||||
|
||||
_ENV_JSON_KEYS = ("GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON", "FILES_CONNECTOR_PROFILES_JSON")
|
||||
_ENV_FILE_KEYS = ("GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE", "FILES_CONNECTOR_PROFILES_FILE")
|
||||
_SUPPORTED_PROVIDERS = {"seafile", "nextcloud", "webdav", "smb", "nfs", "dms", "generic"}
|
||||
_INLINE_SECRET_FIELDS = ("password", "token", "api_key", "access_key", "secret_key")
|
||||
_SUPPORTED_PROVIDERS = {"seafile", "nextcloud", "webdav", "smb", "s3", "sharepoint", "onedrive", "nfs", "dms", "generic"}
|
||||
_INLINE_SECRET_FIELDS = (
|
||||
"password",
|
||||
"token",
|
||||
"api_key",
|
||||
"access_key",
|
||||
"access_key_id",
|
||||
"secret_key",
|
||||
"secret_access_key",
|
||||
"session_token",
|
||||
)
|
||||
|
||||
|
||||
def supported_connector_providers() -> set[str]:
|
||||
@@ -67,15 +80,17 @@ class ConnectorProfile:
|
||||
|
||||
@property
|
||||
def credentials_configured(self) -> bool:
|
||||
if not self.enabled:
|
||||
return False
|
||||
mode = self.credential_mode.casefold()
|
||||
if mode in {"", "none", "anonymous"}:
|
||||
return True
|
||||
if self.secret_ref or self.has_inline_secret or self.password_value or self.token_value:
|
||||
return True
|
||||
if self.token_env:
|
||||
return bool(os.environ.get(self.token_env))
|
||||
return connector_secret_env_available(self.token_env, source_kind=self.source_kind)
|
||||
if self.password_env:
|
||||
return bool(os.environ.get(self.password_env))
|
||||
return connector_secret_env_available(self.password_env, source_kind=self.source_kind)
|
||||
return False
|
||||
|
||||
def to_response(self) -> dict[str, Any]:
|
||||
@@ -97,7 +112,7 @@ class ConnectorProfile:
|
||||
"username": self.username,
|
||||
"capabilities": list(self.capabilities),
|
||||
"policy_sources": [_policy_source_response(source) for source in self.policy_sources],
|
||||
"metadata": dict(self.metadata),
|
||||
"metadata": _response_metadata(self.metadata),
|
||||
"source_kind": self.source_kind,
|
||||
}
|
||||
|
||||
@@ -123,45 +138,21 @@ def connector_profiles_from_payload(value: object) -> list[ConnectorProfile]:
|
||||
|
||||
|
||||
def _profile_from_mapping(value: Mapping[str, Any]) -> ConnectorProfile:
|
||||
profile_id = _clean(value.get("id") or value.get("connector_id") or value.get("name"))
|
||||
if not profile_id:
|
||||
raise ValueError("Connector profiles require id")
|
||||
provider = (_clean(value.get("provider") or value.get("type")) or "generic").casefold()
|
||||
if provider not in _SUPPORTED_PROVIDERS:
|
||||
raise ValueError(f"Unsupported connector provider: {provider}")
|
||||
scope_type = normalize_policy_scope_type(str(value.get("scope_type") or "system"))
|
||||
scope_id = _clean(value.get("scope_id"))
|
||||
if scope_type == "system":
|
||||
scope_id = None
|
||||
elif not scope_id:
|
||||
raise ValueError(f"{scope_type} connector profiles require scope_id")
|
||||
|
||||
credentials = value.get("credentials")
|
||||
credentials = credentials if isinstance(credentials, Mapping) else {}
|
||||
profile_id = _profile_id_from_mapping(value)
|
||||
provider = _provider_from_mapping(value)
|
||||
scope_type, scope_id = _scope_from_mapping(value)
|
||||
credentials = _credentials_from_mapping(value)
|
||||
mode = _clean(value.get("credential_mode") or value.get("auth_type") or credentials.get("mode") or credentials.get("type")) or "none"
|
||||
profile = ConnectorProfile(
|
||||
id=profile_id,
|
||||
label=_clean(value.get("label") or value.get("name")) or profile_id,
|
||||
profile = _profile_from_normalized_mapping(
|
||||
value,
|
||||
profile_id=profile_id,
|
||||
provider=provider,
|
||||
endpoint_url=_clean(value.get("endpoint_url") or value.get("base_url") or value.get("url")),
|
||||
base_path=_clean(value.get("base_path") or value.get("path") or value.get("root")),
|
||||
enabled=_bool(value.get("enabled"), default=True),
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
credential_profile_id=_clean(value.get("credential_profile_id") or value.get("credential_id")),
|
||||
credential_profile_label=_clean(value.get("credential_profile_label") or value.get("credential_label")),
|
||||
credential_mode=mode,
|
||||
username=_clean(value.get("username") or credentials.get("username")),
|
||||
password_env=_clean(value.get("password_env") or credentials.get("password_env")),
|
||||
token_env=_clean(value.get("token_env") or credentials.get("token_env")),
|
||||
secret_ref=_clean(value.get("secret_ref") or credentials.get("secret_ref")),
|
||||
password_value=_clean(value.get("password") or credentials.get("password")),
|
||||
token_value=_clean(value.get("token") or credentials.get("token")),
|
||||
has_inline_secret=any(_clean(value.get(field) or credentials.get(field)) for field in _INLINE_SECRET_FIELDS),
|
||||
capabilities=tuple(_string_list(value.get("capabilities"))),
|
||||
metadata=_public_metadata(value.get("metadata")),
|
||||
credentials=credentials,
|
||||
)
|
||||
return ConnectorProfile(
|
||||
result = ConnectorProfile(
|
||||
id=profile.id,
|
||||
label=profile.label,
|
||||
provider=profile.provider,
|
||||
@@ -185,6 +176,76 @@ def _profile_from_mapping(value: Mapping[str, Any]) -> ConnectorProfile:
|
||||
metadata=profile.metadata,
|
||||
source_kind="settings",
|
||||
)
|
||||
validate_deployment_connector_references(
|
||||
source_kind=result.source_kind,
|
||||
password_env=result.password_env,
|
||||
token_env=result.token_env,
|
||||
metadata=result.metadata,
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def _profile_id_from_mapping(value: Mapping[str, Any]) -> str:
|
||||
profile_id = _clean(value.get("id") or value.get("connector_id") or value.get("name"))
|
||||
if not profile_id:
|
||||
raise ValueError("Connector profiles require id")
|
||||
return profile_id
|
||||
|
||||
|
||||
def _provider_from_mapping(value: Mapping[str, Any]) -> str:
|
||||
provider = (_clean(value.get("provider") or value.get("type")) or "generic").casefold()
|
||||
if provider not in _SUPPORTED_PROVIDERS:
|
||||
raise ValueError(f"Unsupported connector provider: {provider}")
|
||||
return provider
|
||||
|
||||
|
||||
def _scope_from_mapping(value: Mapping[str, Any]) -> tuple[str, str | None]:
|
||||
scope_type = normalize_policy_scope_type(str(value.get("scope_type") or "system"))
|
||||
scope_id = _clean(value.get("scope_id"))
|
||||
if scope_type == "system":
|
||||
return scope_type, None
|
||||
if not scope_id:
|
||||
raise ValueError(f"{scope_type} connector profiles require scope_id")
|
||||
return scope_type, scope_id
|
||||
|
||||
|
||||
def _credentials_from_mapping(value: Mapping[str, Any]) -> Mapping[str, Any]:
|
||||
credentials = value.get("credentials")
|
||||
return credentials if isinstance(credentials, Mapping) else {}
|
||||
|
||||
|
||||
def _profile_from_normalized_mapping(
|
||||
value: Mapping[str, Any],
|
||||
*,
|
||||
profile_id: str,
|
||||
provider: str,
|
||||
scope_type: str,
|
||||
scope_id: str | None,
|
||||
credential_mode: str,
|
||||
credentials: Mapping[str, Any],
|
||||
) -> ConnectorProfile:
|
||||
return ConnectorProfile(
|
||||
id=profile_id,
|
||||
label=_clean(value.get("label") or value.get("name")) or profile_id,
|
||||
provider=provider,
|
||||
endpoint_url=_clean(value.get("endpoint_url") or value.get("base_url") or value.get("url")),
|
||||
base_path=_clean(value.get("base_path") or value.get("path") or value.get("root")),
|
||||
enabled=_bool(value.get("enabled"), default=True),
|
||||
scope_type=scope_type,
|
||||
scope_id=scope_id,
|
||||
credential_profile_id=_clean(value.get("credential_profile_id") or value.get("credential_id")),
|
||||
credential_profile_label=_clean(value.get("credential_profile_label") or value.get("credential_label")),
|
||||
credential_mode=credential_mode,
|
||||
username=_clean(value.get("username") or credentials.get("username")),
|
||||
password_env=_clean(value.get("password_env") or credentials.get("password_env")),
|
||||
token_env=_clean(value.get("token_env") or credentials.get("token_env")),
|
||||
secret_ref=_clean(value.get("secret_ref") or credentials.get("secret_ref")),
|
||||
password_value=_clean(value.get("password") or credentials.get("password")),
|
||||
token_value=_clean(value.get("token") or credentials.get("token")),
|
||||
has_inline_secret=any(_clean(value.get(field) or credentials.get(field)) for field in _INLINE_SECRET_FIELDS),
|
||||
capabilities=tuple(_string_list(value.get("capabilities"))),
|
||||
metadata=_public_metadata(value.get("metadata")),
|
||||
)
|
||||
|
||||
|
||||
def _raw_profiles_payload(settings: object | None) -> object:
|
||||
@@ -243,6 +304,16 @@ def _public_metadata(value: object) -> Mapping[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
def _response_metadata(value: Mapping[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
str(key): item
|
||||
for key, item in value.items()
|
||||
if str(key).strip().casefold() not in _INLINE_SECRET_FIELDS
|
||||
and not str(key).strip().casefold().endswith("_env")
|
||||
and str(key).strip().casefold() != "ca_bundle"
|
||||
}
|
||||
|
||||
|
||||
def _string_list(value: object) -> list[str]:
|
||||
if value is None:
|
||||
return []
|
||||
|
||||
@@ -110,7 +110,52 @@ def connector_provider_descriptors() -> tuple[ConnectorProviderDescriptor, ...]:
|
||||
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling after download.",
|
||||
preview_strategy="Previews are generated from the frozen managed file after import or sync, not directly from the share.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Requires the optional smbprotocol dependency and an smb://server[:port]/share[/path] profile endpoint.",
|
||||
notes="Browse/import logic is implemented, but live smbprotocol access fails closed until the SDK supports connection-time DNS/IP pinning for initial connections and DFS referrals.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="s3",
|
||||
label="S3-compatible object storage",
|
||||
protocol="s3-api",
|
||||
implemented=True,
|
||||
browse_supported=True,
|
||||
import_supported=True,
|
||||
optional_dependency="boto3",
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="On-demand bucket/prefix browse and object import/sync; sync updates managed versions and never mutates the remote bucket.",
|
||||
conflict_strategy="Managed import/sync uses existing files conflict_strategy handling after object download.",
|
||||
preview_strategy="Previews are generated from the frozen managed file after import or sync, not directly from the bucket.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Browse/import logic is implemented, but live boto3 access fails closed until its HTTP transport supports connection-time DNS/IP pinning and redirect revalidation.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="sharepoint",
|
||||
label="SharePoint",
|
||||
protocol="microsoft-graph/sharepoint",
|
||||
implemented=False,
|
||||
browse_supported=False,
|
||||
import_supported=False,
|
||||
optional_dependency=None,
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="Planned read-only browse/import through deployment-managed Microsoft Graph or SharePoint credentials.",
|
||||
conflict_strategy=freeze_model,
|
||||
preview_strategy="Will preview from frozen managed file after import, not directly from SharePoint.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Provider/profile key is reserved; live browsing/import still needs Graph/SharePoint authentication and paging implementation.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="onedrive",
|
||||
label="OneDrive",
|
||||
protocol="microsoft-graph/onedrive",
|
||||
implemented=False,
|
||||
browse_supported=False,
|
||||
import_supported=False,
|
||||
optional_dependency=None,
|
||||
permission_model=governed_permissions,
|
||||
sync_strategy="Planned read-only browse/import through deployment-managed Microsoft Graph credentials.",
|
||||
conflict_strategy=freeze_model,
|
||||
preview_strategy="Will preview from frozen managed file after import, not directly from OneDrive.",
|
||||
audit_events=("files.connector.imported", "files.connector.synced", "files.connector.accessed"),
|
||||
notes="Provider/profile key is reserved; live browsing/import still needs Graph drive selection, OAuth/app registration, and paging implementation.",
|
||||
),
|
||||
ConnectorProviderDescriptor(
|
||||
provider="nfs",
|
||||
|
||||
239
src/govoplan_files/backend/storage/http_client.py
Normal file
239
src/govoplan_files/backend/storage/http_client.py
Normal file
@@ -0,0 +1,239 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import socket
|
||||
import ssl
|
||||
import select
|
||||
from collections.abc import Mapping
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Iterator
|
||||
|
||||
import httpcore
|
||||
import httpx
|
||||
|
||||
from govoplan_core.security.outbound_http import (
|
||||
OutboundHttpError,
|
||||
bounded_chunks_bytes,
|
||||
create_outbound_connection,
|
||||
response_limit,
|
||||
validate_outbound_http_url,
|
||||
)
|
||||
|
||||
|
||||
class ConnectorHttpError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConnectorHttpResponse:
|
||||
status_code: int
|
||||
headers: Mapping[str, str]
|
||||
content: bytes
|
||||
|
||||
|
||||
_HTTPCORE_TRANSPORT_ERRORS = (
|
||||
httpcore.TimeoutException,
|
||||
httpcore.NetworkError,
|
||||
httpcore.ProtocolError,
|
||||
httpcore.ProxyError,
|
||||
httpcore.UnsupportedProtocol,
|
||||
)
|
||||
|
||||
|
||||
class _SocketNetworkStream(httpcore.NetworkStream):
|
||||
"""Public httpcore NetworkStream adapter around an approved socket."""
|
||||
|
||||
def __init__(self, sock: socket.socket) -> None:
|
||||
self._socket = sock
|
||||
|
||||
def read(self, max_bytes: int, timeout: float | None = None) -> bytes:
|
||||
try:
|
||||
self._socket.settimeout(timeout)
|
||||
return self._socket.recv(max_bytes)
|
||||
except socket.timeout as exc:
|
||||
raise httpcore.ReadTimeout(str(exc)) from exc
|
||||
except OSError as exc:
|
||||
raise httpcore.ReadError(str(exc)) from exc
|
||||
|
||||
def write(self, buffer: bytes, timeout: float | None = None) -> None:
|
||||
try:
|
||||
self._socket.settimeout(timeout)
|
||||
self._socket.sendall(buffer)
|
||||
except socket.timeout as exc:
|
||||
raise httpcore.WriteTimeout(str(exc)) from exc
|
||||
except OSError as exc:
|
||||
raise httpcore.WriteError(str(exc)) from exc
|
||||
|
||||
def close(self) -> None:
|
||||
self._socket.close()
|
||||
|
||||
def start_tls(
|
||||
self,
|
||||
ssl_context: ssl.SSLContext,
|
||||
server_hostname: str | None = None,
|
||||
timeout: float | None = None,
|
||||
) -> httpcore.NetworkStream:
|
||||
try:
|
||||
self._socket.settimeout(timeout)
|
||||
tls_socket = ssl_context.wrap_socket(self._socket, server_hostname=server_hostname)
|
||||
except socket.timeout as exc:
|
||||
self.close()
|
||||
raise httpcore.ConnectTimeout(str(exc)) from exc
|
||||
except OSError as exc:
|
||||
self.close()
|
||||
raise httpcore.ConnectError(str(exc)) from exc
|
||||
return _SocketNetworkStream(tls_socket)
|
||||
|
||||
def get_extra_info(self, info: str) -> Any:
|
||||
if info == "ssl_object" and isinstance(self._socket, ssl.SSLSocket):
|
||||
return self._socket
|
||||
if info == "client_addr":
|
||||
return self._socket.getsockname()
|
||||
if info == "server_addr":
|
||||
return self._socket.getpeername()
|
||||
if info == "socket":
|
||||
return self._socket
|
||||
if info == "is_readable":
|
||||
try:
|
||||
return bool(select.select([self._socket], [], [], 0)[0])
|
||||
except (OSError, ValueError):
|
||||
return True
|
||||
return None
|
||||
|
||||
|
||||
class _OutboundPolicyNetworkBackend(httpcore.NetworkBackend):
|
||||
def connect_tcp(
|
||||
self,
|
||||
host: str,
|
||||
port: int,
|
||||
timeout: float | None = None,
|
||||
local_address: str | None = None,
|
||||
socket_options: Any = None,
|
||||
) -> httpcore.NetworkStream:
|
||||
source_address = None if local_address is None else (local_address, 0)
|
||||
try:
|
||||
sock = create_outbound_connection(
|
||||
host,
|
||||
port,
|
||||
timeout=timeout,
|
||||
source_address=source_address,
|
||||
socket_options=socket_options,
|
||||
label="File connector HTTP endpoint",
|
||||
)
|
||||
except socket.timeout as exc:
|
||||
raise httpcore.ConnectTimeout(str(exc)) from exc
|
||||
except (OSError, OutboundHttpError) as exc:
|
||||
raise httpcore.ConnectError(str(exc)) from exc
|
||||
return _SocketNetworkStream(sock)
|
||||
|
||||
def connect_unix_socket(self, path: str, timeout: float | None = None, socket_options: Any = None): # type: ignore[no-untyped-def]
|
||||
del path, timeout, socket_options
|
||||
raise httpcore.ConnectError("Unix sockets are not supported for file connectors")
|
||||
|
||||
|
||||
class _HttpcoreResponseStream(httpx.SyncByteStream):
|
||||
def __init__(self, stream: Any, *, request: httpx.Request) -> None:
|
||||
self._stream = stream
|
||||
self._request = request
|
||||
|
||||
def __iter__(self): # type: ignore[no-untyped-def]
|
||||
try:
|
||||
yield from self._stream
|
||||
except _HTTPCORE_TRANSPORT_ERRORS as exc:
|
||||
raise httpx.TransportError(str(exc), request=self._request) from exc
|
||||
|
||||
def close(self) -> None:
|
||||
if hasattr(self._stream, "close"):
|
||||
self._stream.close()
|
||||
|
||||
|
||||
class _OutboundPolicyHTTPTransport(httpx.BaseTransport):
|
||||
def __init__(self) -> None:
|
||||
self._connection_pool = httpcore.ConnectionPool(
|
||||
ssl_context=httpx.create_ssl_context(verify=True, trust_env=False),
|
||||
max_connections=100,
|
||||
max_keepalive_connections=20,
|
||||
keepalive_expiry=5.0,
|
||||
network_backend=_OutboundPolicyNetworkBackend(),
|
||||
)
|
||||
|
||||
def handle_request(self, request: httpx.Request) -> httpx.Response:
|
||||
core_request = httpcore.Request(
|
||||
method=request.method,
|
||||
url=httpcore.URL(
|
||||
scheme=request.url.raw_scheme,
|
||||
host=request.url.raw_host,
|
||||
port=request.url.port,
|
||||
target=request.url.raw_path,
|
||||
),
|
||||
headers=request.headers.raw,
|
||||
content=request.stream,
|
||||
extensions=request.extensions,
|
||||
)
|
||||
try:
|
||||
response = self._connection_pool.handle_request(core_request)
|
||||
except _HTTPCORE_TRANSPORT_ERRORS as exc:
|
||||
raise httpx.TransportError(str(exc), request=request) from exc
|
||||
return httpx.Response(
|
||||
status_code=response.status,
|
||||
headers=response.headers,
|
||||
stream=_HttpcoreResponseStream(response.stream, request=request),
|
||||
extensions=response.extensions,
|
||||
)
|
||||
|
||||
def close(self) -> None:
|
||||
self._connection_pool.close()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _stream_connector_request(method: str, url: str, **kwargs: Any) -> Iterator[httpx.Response]:
|
||||
with httpx.Client(
|
||||
transport=_OutboundPolicyHTTPTransport(),
|
||||
follow_redirects=False,
|
||||
timeout=kwargs.pop("timeout", 15.0),
|
||||
) as client:
|
||||
with client.stream(method, url, **kwargs) as response:
|
||||
yield response
|
||||
|
||||
|
||||
def request_connector_bytes(
|
||||
method: str,
|
||||
url: str,
|
||||
*,
|
||||
headers: Mapping[str, str] | None = None,
|
||||
params: Mapping[str, str] | None = None,
|
||||
data: Mapping[str, str] | bytes | str | None = None,
|
||||
content: bytes | str | None = None,
|
||||
auth: Any = None,
|
||||
timeout: float = 15.0,
|
||||
kind: str = "structured",
|
||||
max_bytes: int | None = None,
|
||||
label: str = "File connector",
|
||||
) -> ConnectorHttpResponse:
|
||||
try:
|
||||
validated_url = validate_outbound_http_url(url, label=f"{label} URL")
|
||||
effective_limit = response_limit(kind) if max_bytes is None else min(int(max_bytes), response_limit(kind))
|
||||
with _stream_connector_request(
|
||||
method,
|
||||
validated_url,
|
||||
headers=dict(headers or {}),
|
||||
params=params,
|
||||
data=data,
|
||||
content=content,
|
||||
auth=auth,
|
||||
timeout=timeout,
|
||||
) as response:
|
||||
body = bounded_chunks_bytes(
|
||||
response.iter_bytes(),
|
||||
headers=response.headers,
|
||||
max_bytes=effective_limit,
|
||||
kind=kind,
|
||||
label=f"{label} response",
|
||||
)
|
||||
return ConnectorHttpResponse(
|
||||
status_code=response.status_code,
|
||||
headers=dict(response.headers),
|
||||
content=body,
|
||||
)
|
||||
except (httpx.HTTPError, OutboundHttpError, ValueError) as exc:
|
||||
raise ConnectorHttpError(str(exc)) from exc
|
||||
@@ -15,21 +15,6 @@ from govoplan_files.backend.storage.common import (
|
||||
utcnow,
|
||||
)
|
||||
from govoplan_files.backend.storage.files import (
|
||||
_active_asset_at_path,
|
||||
_active_asset_exists,
|
||||
_asset_owner_id,
|
||||
_asset_query_for_owner,
|
||||
_candidate_renamed_path,
|
||||
_copy_asset_to_path,
|
||||
_get_or_create_blob,
|
||||
_next_available_logical_path,
|
||||
_normalize_conflict_strategy,
|
||||
_resolution_by_path,
|
||||
_soft_delete_conflicting_asset,
|
||||
_split_logical_path,
|
||||
_storage_backend_name,
|
||||
_storage_bucket_name,
|
||||
_storage_key,
|
||||
asset_is_audit_relevant,
|
||||
create_file_asset,
|
||||
current_version_and_blob,
|
||||
@@ -42,10 +27,6 @@ from govoplan_files.backend.storage.files import (
|
||||
soft_delete_assets,
|
||||
)
|
||||
from govoplan_files.backend.storage.folders import (
|
||||
_active_folder_exists,
|
||||
_ensure_target_folder_hierarchy,
|
||||
_folder_query_for_owner,
|
||||
_owner_filter,
|
||||
create_folder,
|
||||
list_folders_for_user,
|
||||
soft_delete_folder,
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -21,6 +21,7 @@ GROUP_ID = "group-1"
|
||||
class FilesAccessProviderTests(unittest.TestCase):
|
||||
def test_file_access_provider_explains_owner_share_admin_and_missing_resources(self) -> None:
|
||||
session = _session()
|
||||
self.addCleanup(_close_session, session)
|
||||
_seed_access_subjects(session)
|
||||
owned = FileAsset(id="file-owned", tenant_id=TENANT_ID, owner_type="user", owner_user_id=USER_ID, display_path="owned.pdf", filename="owned.pdf")
|
||||
shared = FileAsset(id="file-shared", tenant_id=TENANT_ID, owner_type="user", owner_user_id=OTHER_USER_ID, display_path="shared.pdf", filename="shared.pdf")
|
||||
@@ -46,6 +47,7 @@ class FilesAccessProviderTests(unittest.TestCase):
|
||||
|
||||
def test_file_access_provider_explains_persisted_and_virtual_folders(self) -> None:
|
||||
session = _session()
|
||||
self.addCleanup(_close_session, session)
|
||||
_seed_access_subjects(session)
|
||||
persisted = FileFolder(id="folder-persisted", tenant_id=TENANT_ID, owner_type="group", owner_group_id=GROUP_ID, path="records")
|
||||
virtual_child = FileAsset(
|
||||
@@ -80,6 +82,12 @@ def _session():
|
||||
return sessionmaker(bind=engine, future=True)()
|
||||
|
||||
|
||||
def _close_session(session) -> None:
|
||||
engine = session.get_bind()
|
||||
session.close()
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def _seed_access_subjects(session) -> None:
|
||||
session.add_all([
|
||||
Account(id="account-1", email="one@example.test", normalized_email="one@example.test"),
|
||||
|
||||
272
tests/test_connector_credential_deletion.py
Normal file
272
tests/test_connector_credential_deletion.py
Normal file
@@ -0,0 +1,272 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import patch
|
||||
|
||||
from sqlalchemy import create_engine, inspect
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from govoplan_access.backend.db import models as access_models # noqa: F401 - resolve Files user foreign keys
|
||||
from govoplan_core.core.change_sequence import ChangeSequenceEntry
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_core.security.secrets import encrypt_secret
|
||||
from govoplan_files.backend.db.models import FileConnectorCredential, FileConnectorProfile
|
||||
from govoplan_files.backend.router import deactivate_connector_credential, deactivate_connector_profile
|
||||
|
||||
|
||||
class Principal:
|
||||
tenant_id = "tenant-1"
|
||||
user = SimpleNamespace(id="user-1")
|
||||
api_key = None
|
||||
|
||||
def has(self, scope: str) -> bool:
|
||||
return scope == "files:file:admin"
|
||||
|
||||
|
||||
class ConnectorCredentialDeletionTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.engine = create_engine("sqlite:///:memory:")
|
||||
Base.metadata.create_all(
|
||||
self.engine,
|
||||
tables=[
|
||||
FileConnectorCredential.__table__,
|
||||
FileConnectorProfile.__table__,
|
||||
ChangeSequenceEntry.__table__,
|
||||
],
|
||||
)
|
||||
self.session = sessionmaker(bind=self.engine)()
|
||||
self.principal = Principal()
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.session.close()
|
||||
Base.metadata.drop_all(bind=self.engine)
|
||||
self.engine.dispose()
|
||||
|
||||
def test_delete_credential_scrubs_material_audits_and_disables_dependents(self) -> None:
|
||||
credential = self._credential()
|
||||
profile = self._profile(credential_profile_id=credential.id)
|
||||
self.session.add_all([credential, profile])
|
||||
self.session.commit()
|
||||
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
|
||||
) as audit:
|
||||
response = deactivate_connector_credential(
|
||||
credential.id,
|
||||
session=self.session,
|
||||
principal=self.principal, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
self.assertFalse(response.enabled)
|
||||
self.assertFalse(response.credentials_configured)
|
||||
self.assertIsNone(response.credential_secret_source)
|
||||
self.session.refresh(credential)
|
||||
self.session.refresh(profile)
|
||||
self._assert_credential_scrubbed(credential)
|
||||
self._assert_profile_scrubbed(profile)
|
||||
|
||||
calls = {call.kwargs["action"]: call.kwargs for call in audit.call_args_list}
|
||||
self.assertEqual(
|
||||
{"files.connector_profile_deleted", "files.connector_credential_deleted"},
|
||||
set(calls),
|
||||
)
|
||||
credential_audit = calls["files.connector_credential_deleted"]
|
||||
self.assertEqual("encrypted_database", credential_audit["details"]["storage_backend"])
|
||||
self.assertEqual(["password", "token"], credential_audit["details"]["deleted_secret_kinds"])
|
||||
self.assertEqual(1, credential_audit["details"]["affected_profile_count"])
|
||||
self.assertNotIn("do-not-audit", repr(audit.call_args_list))
|
||||
|
||||
changes = self.session.query(ChangeSequenceEntry).order_by(ChangeSequenceEntry.id.asc()).all()
|
||||
self.assertEqual([credential.id, profile.id], [change.resource_id for change in changes])
|
||||
self.assertEqual(["deleted", "updated"], [change.operation for change in changes])
|
||||
|
||||
def test_delete_profile_scrubs_inline_credentials_and_audits(self) -> None:
|
||||
profile = self._profile(credential_profile_id="shared-credential")
|
||||
self.session.add(profile)
|
||||
self.session.commit()
|
||||
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
|
||||
) as audit:
|
||||
response = deactivate_connector_profile(
|
||||
profile.id,
|
||||
session=self.session,
|
||||
principal=self.principal, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
self.assertFalse(response.enabled)
|
||||
self.assertFalse(response.credentials_configured)
|
||||
self.assertIsNone(response.credential_profile_id)
|
||||
self.session.refresh(profile)
|
||||
self._assert_profile_scrubbed(profile)
|
||||
audit.assert_called_once()
|
||||
call = audit.call_args.kwargs
|
||||
self.assertEqual("files.connector_profile_deleted", call["action"])
|
||||
self.assertEqual("api_delete", call["details"]["deletion_reason"])
|
||||
self.assertIn("credential_profile", call["details"]["removed_reference_kinds"])
|
||||
self.assertNotIn("do-not-audit", repr(call))
|
||||
|
||||
def test_delete_rolls_back_when_audit_fails(self) -> None:
|
||||
credential = self._credential()
|
||||
self.session.add(credential)
|
||||
self.session.commit()
|
||||
original_password = credential.password_encrypted
|
||||
original_token = credential.token_encrypted
|
||||
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.connector_credential_deletion.audit_event",
|
||||
side_effect=RuntimeError("audit unavailable"),
|
||||
), self.assertRaisesRegex(RuntimeError, "audit unavailable"):
|
||||
deactivate_connector_credential(
|
||||
credential.id,
|
||||
session=self.session,
|
||||
principal=self.principal, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
persisted = self.session.get(FileConnectorCredential, credential.id)
|
||||
assert persisted is not None
|
||||
self.assertTrue(persisted.enabled)
|
||||
self.assertEqual(original_password, persisted.password_encrypted)
|
||||
self.assertEqual(original_token, persisted.token_encrypted)
|
||||
self.assertEqual(0, self.session.query(ChangeSequenceEntry).count())
|
||||
|
||||
def test_delete_detaches_unowned_legacy_secret_reference_without_claiming_provider_deletion(self) -> None:
|
||||
credential = self._credential(secret_ref="vault:tenant-1:files:credential")
|
||||
self.session.add(credential)
|
||||
self.session.commit()
|
||||
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
|
||||
) as audit:
|
||||
response = deactivate_connector_credential(
|
||||
credential.id,
|
||||
session=self.session,
|
||||
principal=self.principal, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
self.assertFalse(response.enabled)
|
||||
persisted = self.session.get(FileConnectorCredential, credential.id)
|
||||
assert persisted is not None
|
||||
self._assert_credential_scrubbed(persisted)
|
||||
audit.assert_called_once()
|
||||
details = audit.call_args.kwargs["details"]
|
||||
self.assertEqual("unowned_external_reference_detached", details["storage_backend"])
|
||||
self.assertIn("unowned_external_secret_ref", details["removed_reference_kinds"])
|
||||
self.assertNotIn("vault:tenant-1:files:credential", repr(audit.call_args))
|
||||
|
||||
def test_retirement_scrubs_and_audits_before_tables_are_dropped(self) -> None:
|
||||
from govoplan_files.backend.manifest import manifest
|
||||
|
||||
self.session.add_all([self._credential(), self._profile()])
|
||||
self.session.commit()
|
||||
retirement_provider = manifest.migration_spec.retirement_provider
|
||||
assert retirement_provider is not None
|
||||
plan = retirement_provider(self.session, "files")
|
||||
assert plan.destroy_data_executor is not None
|
||||
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
|
||||
) as audit:
|
||||
plan.destroy_data_executor(self.session, "files")
|
||||
|
||||
self.assertEqual(2, audit.call_count)
|
||||
self.assertEqual(
|
||||
{"module_data_retired"},
|
||||
{call.kwargs["details"]["deletion_reason"] for call in audit.call_args_list},
|
||||
)
|
||||
self.assertNotIn("do-not-audit", repr(audit.call_args_list))
|
||||
self.assertFalse(inspect(self.engine).has_table(FileConnectorCredential.__tablename__))
|
||||
self.assertFalse(inspect(self.engine).has_table(FileConnectorProfile.__tablename__))
|
||||
|
||||
def test_retirement_detaches_unowned_external_reference_before_drop(self) -> None:
|
||||
from govoplan_files.backend.manifest import manifest
|
||||
|
||||
self.session.add(self._credential(secret_ref="vault:tenant-1:files:credential"))
|
||||
self.session.commit()
|
||||
retirement_provider = manifest.migration_spec.retirement_provider
|
||||
assert retirement_provider is not None
|
||||
plan = retirement_provider(self.session, "files")
|
||||
assert plan.destroy_data_executor is not None
|
||||
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.connector_credential_deletion.audit_event"
|
||||
) as audit:
|
||||
plan.destroy_data_executor(self.session, "files")
|
||||
|
||||
audit.assert_called_once()
|
||||
details = audit.call_args.kwargs["details"]
|
||||
self.assertIn("unowned_external_secret_ref", details["removed_reference_kinds"])
|
||||
self.assertNotIn("vault:tenant-1:files:credential", repr(audit.call_args))
|
||||
self.assertFalse(inspect(self.engine).has_table(FileConnectorCredential.__tablename__))
|
||||
self.assertFalse(inspect(self.engine).has_table(FileConnectorProfile.__tablename__))
|
||||
|
||||
@staticmethod
|
||||
def _credential(*, secret_ref: str | None = None) -> FileConnectorCredential:
|
||||
return FileConnectorCredential(
|
||||
id="credential-1",
|
||||
tenant_id="tenant-1",
|
||||
scope_type="tenant",
|
||||
scope_id="tenant-1",
|
||||
label="Credential",
|
||||
provider="webdav",
|
||||
enabled=True,
|
||||
credential_mode="basic",
|
||||
username="connector-user",
|
||||
password_encrypted=encrypt_secret("do-not-audit-password"),
|
||||
token_encrypted=encrypt_secret("do-not-audit-token"),
|
||||
password_env="DEPLOYMENT_PASSWORD",
|
||||
token_env="DEPLOYMENT_TOKEN",
|
||||
secret_ref=secret_ref,
|
||||
policy={},
|
||||
metadata_={"private_hint": "do-not-audit-metadata"},
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _profile(*, credential_profile_id: str | None = None) -> FileConnectorProfile:
|
||||
return FileConnectorProfile(
|
||||
id="profile-1",
|
||||
tenant_id="tenant-1",
|
||||
scope_type="tenant",
|
||||
scope_id="tenant-1",
|
||||
label="Profile",
|
||||
provider="webdav",
|
||||
endpoint_url="https://dav.example.test",
|
||||
enabled=True,
|
||||
credential_profile_id=credential_profile_id,
|
||||
credential_mode="basic",
|
||||
username="connector-user",
|
||||
password_encrypted=encrypt_secret("do-not-audit-profile-password"),
|
||||
token_encrypted=encrypt_secret("do-not-audit-profile-token"),
|
||||
password_env="DEPLOYMENT_PROFILE_PASSWORD",
|
||||
token_env="DEPLOYMENT_PROFILE_TOKEN",
|
||||
policy={},
|
||||
metadata_={"private_hint": "do-not-audit-profile-metadata"},
|
||||
)
|
||||
|
||||
def _assert_credential_scrubbed(self, row: FileConnectorCredential) -> None:
|
||||
self.assertFalse(row.enabled)
|
||||
self.assertEqual("none", row.credential_mode)
|
||||
self.assertIsNone(row.username)
|
||||
self.assertIsNone(row.password_encrypted)
|
||||
self.assertIsNone(row.token_encrypted)
|
||||
self.assertIsNone(row.password_env)
|
||||
self.assertIsNone(row.token_env)
|
||||
self.assertIsNone(row.secret_ref)
|
||||
self.assertEqual({}, row.metadata_)
|
||||
|
||||
def _assert_profile_scrubbed(self, row: FileConnectorProfile) -> None:
|
||||
self.assertFalse(row.enabled)
|
||||
self.assertIsNone(row.credential_profile_id)
|
||||
self.assertEqual("none", row.credential_mode)
|
||||
self.assertIsNone(row.username)
|
||||
self.assertIsNone(row.password_encrypted)
|
||||
self.assertIsNone(row.token_encrypted)
|
||||
self.assertIsNone(row.password_env)
|
||||
self.assertIsNone(row.token_env)
|
||||
self.assertIsNone(row.secret_ref)
|
||||
self.assertEqual({}, row.metadata_)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
227
tests/test_connector_deployment.py
Normal file
227
tests/test_connector_deployment.py
Normal file
@@ -0,0 +1,227 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import patch
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from fastapi import HTTPException
|
||||
|
||||
from govoplan_files.backend.router import discover_connector_endpoint
|
||||
from govoplan_files.backend.schemas import FileConnectorDiscoveryRequest
|
||||
from govoplan_files.backend.storage.connector_browse import ConnectorBrowseError, _profile_password, _s3_verify
|
||||
from govoplan_files.backend.storage.connector_deployment import (
|
||||
ConnectorDeploymentConfigurationError,
|
||||
connector_effective_endpoint_url,
|
||||
connector_secret_env_value,
|
||||
reject_api_controlled_deployment_references,
|
||||
)
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile
|
||||
from govoplan_files.backend.storage.connector_profile_store import create_connector_profile_row
|
||||
|
||||
|
||||
class ConnectorDeploymentBoundaryTests(unittest.TestCase):
|
||||
def test_database_profile_cannot_read_arbitrary_process_environment(self) -> None:
|
||||
profile = ConnectorProfile(
|
||||
id="tenant-webdav",
|
||||
label="Tenant WebDAV",
|
||||
provider="webdav",
|
||||
password_env="MASTER_KEY_B64",
|
||||
source_kind="database",
|
||||
)
|
||||
with patch.dict(
|
||||
os.environ,
|
||||
{
|
||||
"MASTER_KEY_B64": "must-not-leave-process",
|
||||
"GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST": "MASTER_KEY_B64",
|
||||
},
|
||||
clear=False,
|
||||
), self.assertRaisesRegex(ConnectorBrowseError, "deployment-owned"):
|
||||
_profile_password(profile)
|
||||
|
||||
def test_deployment_profile_requires_exact_secret_allowlist(self) -> None:
|
||||
with patch.dict(
|
||||
os.environ,
|
||||
{
|
||||
"GOVOPLAN_FILES_WEBDAV_PASSWORD": "deployment-secret",
|
||||
"GOVOPLAN_CONNECTOR_SECRET_ENV_ALLOWLIST": "GOVOPLAN_FILES_WEBDAV_PASSWORD",
|
||||
},
|
||||
clear=False,
|
||||
):
|
||||
self.assertEqual(
|
||||
"deployment-secret",
|
||||
connector_secret_env_value(
|
||||
"GOVOPLAN_FILES_WEBDAV_PASSWORD",
|
||||
source_kind="settings",
|
||||
),
|
||||
)
|
||||
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "not listed"):
|
||||
connector_secret_env_value("MASTER_KEY_B64", source_kind="settings")
|
||||
|
||||
def test_api_profiles_cannot_select_secret_environment_names(self) -> None:
|
||||
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "deployment-owned"):
|
||||
reject_api_controlled_deployment_references(password_env="DATABASE_URL")
|
||||
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "deployment-owned"):
|
||||
reject_api_controlled_deployment_references(metadata={"secret_access_key_env": "MASTER_KEY_B64"})
|
||||
|
||||
session = MagicMock()
|
||||
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "deployment-owned"):
|
||||
create_connector_profile_row(
|
||||
session,
|
||||
tenant_id="tenant-1",
|
||||
user_id="user-1",
|
||||
profile_id="unsafe",
|
||||
label="Unsafe",
|
||||
provider="webdav",
|
||||
scope_type="tenant",
|
||||
password_env="DATABASE_URL",
|
||||
)
|
||||
session.get.assert_not_called()
|
||||
|
||||
def test_api_profiles_cannot_create_unowned_external_secret_references(self) -> None:
|
||||
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "provider ownership"):
|
||||
reject_api_controlled_deployment_references(secret_ref="vault:tenant-1:files")
|
||||
|
||||
session = MagicMock()
|
||||
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "provider ownership"):
|
||||
create_connector_profile_row(
|
||||
session,
|
||||
tenant_id="tenant-1",
|
||||
user_id="user-1",
|
||||
profile_id="unsafe-secret-ref",
|
||||
label="Unsafe secret reference",
|
||||
provider="webdav",
|
||||
scope_type="tenant",
|
||||
credential_mode="secret_ref",
|
||||
secret_ref="vault:tenant-1:files",
|
||||
)
|
||||
session.get.assert_not_called()
|
||||
|
||||
def test_api_profiles_cannot_hide_plaintext_credentials_in_metadata(self) -> None:
|
||||
for metadata in (
|
||||
{"secret_access_key": "plaintext-secret"},
|
||||
{"credentials": {"password": "plaintext-secret"}},
|
||||
{"provider": {"refresh_token": "plaintext-secret"}},
|
||||
):
|
||||
with self.subTest(metadata=metadata), self.assertRaisesRegex(
|
||||
ConnectorDeploymentConfigurationError,
|
||||
"dedicated encrypted credential fields",
|
||||
):
|
||||
reject_api_controlled_deployment_references(metadata=metadata)
|
||||
|
||||
def test_profile_responses_hide_environment_and_local_ca_references(self) -> None:
|
||||
profile = ConnectorProfile(
|
||||
id="deployment-s3",
|
||||
label="Deployment S3",
|
||||
provider="s3",
|
||||
metadata={
|
||||
"bucket": "documents",
|
||||
"secret_access_key_env": "GOVOPLAN_FILES_S3_SECRET",
|
||||
"ca_bundle": "/etc/govoplan/connector-ca.pem",
|
||||
},
|
||||
)
|
||||
|
||||
self.assertEqual({"bucket": "documents"}, profile.to_response()["metadata"])
|
||||
|
||||
def test_ca_bundle_must_be_an_exact_deployment_allowlisted_file(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
allowed = Path(temp_dir, "connector-ca.pem")
|
||||
other = Path(temp_dir, "other-ca.pem")
|
||||
allowed.write_text("test CA", encoding="utf-8")
|
||||
other.write_text("other CA", encoding="utf-8")
|
||||
with patch.dict(
|
||||
os.environ,
|
||||
{
|
||||
"APP_ENV": "production",
|
||||
"GOVOPLAN_CONNECTOR_CA_BUNDLE_ALLOWLIST": str(allowed),
|
||||
},
|
||||
clear=False,
|
||||
):
|
||||
profile = ConnectorProfile(
|
||||
id="s3",
|
||||
label="S3",
|
||||
provider="s3",
|
||||
metadata={"ca_bundle": str(allowed)},
|
||||
)
|
||||
self.assertEqual(str(allowed.resolve()), _s3_verify(profile))
|
||||
with self.assertRaisesRegex(ConnectorDeploymentConfigurationError, "not listed"):
|
||||
reject_api_controlled_deployment_references(metadata={"ca_bundle": str(other)})
|
||||
|
||||
def test_tls_verification_can_only_be_disabled_in_development(self) -> None:
|
||||
with patch.dict(os.environ, {"APP_ENV": "production"}, clear=False), self.assertRaisesRegex(
|
||||
ConnectorDeploymentConfigurationError,
|
||||
"dev/test",
|
||||
):
|
||||
reject_api_controlled_deployment_references(metadata={"verify_tls": False})
|
||||
with patch.dict(os.environ, {"APP_ENV": "test"}, clear=False):
|
||||
reject_api_controlled_deployment_references(metadata={"verify_tls": False})
|
||||
|
||||
def test_effective_endpoint_uses_webdav_override(self) -> None:
|
||||
self.assertEqual(
|
||||
"https://dav.example.test/root",
|
||||
connector_effective_endpoint_url(
|
||||
provider="seafile",
|
||||
endpoint_url="https://seafile.example.test",
|
||||
metadata={"webdav_endpoint_url": "https://dav.example.test/root"},
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class ConnectorDiscoveryBoundaryTests(unittest.TestCase):
|
||||
@staticmethod
|
||||
def _principal() -> SimpleNamespace:
|
||||
return SimpleNamespace(tenant_id="tenant-1", user=SimpleNamespace(id="user-1"), api_key=None)
|
||||
|
||||
def test_discovery_rejects_environment_credentials_before_io(self) -> None:
|
||||
payload = FileConnectorDiscoveryRequest(
|
||||
provider="webdav",
|
||||
endpoint_url="https://dav.example.test",
|
||||
credential_mode="basic",
|
||||
credentials={"username": "admin", "password_env": "MASTER_KEY_B64"},
|
||||
)
|
||||
with patch("govoplan_files.backend.router.browse_connector_profile") as browse, self.assertRaises(
|
||||
HTTPException
|
||||
) as raised:
|
||||
discover_connector_endpoint(payload, session=object(), principal=self._principal()) # type: ignore[arg-type]
|
||||
self.assertEqual(400, raised.exception.status_code)
|
||||
browse.assert_not_called()
|
||||
|
||||
def test_discovery_applies_policy_and_audit_before_each_io_candidate(self) -> None:
|
||||
payload = FileConnectorDiscoveryRequest(
|
||||
provider="webdav",
|
||||
endpoint_url="https://dav.example.test/root",
|
||||
metadata={
|
||||
"webdav_endpoint_url": "https://bypass.example.test",
|
||||
"static_listing": {"": []},
|
||||
},
|
||||
)
|
||||
events: list[str] = []
|
||||
|
||||
def ensure(*_args: object, **kwargs: object) -> None:
|
||||
self.assertEqual("https://dav.example.test/root/", kwargs["endpoint_url"])
|
||||
events.append("policy")
|
||||
|
||||
def audit(*_args: object, **_kwargs: object) -> None:
|
||||
events.append("audit")
|
||||
|
||||
def browse(profile: ConnectorProfile, **_kwargs: object) -> list[object]:
|
||||
self.assertEqual({}, profile.metadata)
|
||||
events.append("io")
|
||||
return []
|
||||
|
||||
with patch("govoplan_files.backend.router._ensure_connector_configuration_allowed", side_effect=ensure), patch(
|
||||
"govoplan_files.backend.router._audit_connector_discovery_attempt",
|
||||
side_effect=audit,
|
||||
), patch("govoplan_files.backend.router.browse_connector_profile", side_effect=browse):
|
||||
response = discover_connector_endpoint(payload, session=object(), principal=self._principal()) # type: ignore[arg-type]
|
||||
|
||||
self.assertEqual("usable", response.status)
|
||||
self.assertEqual(["policy", "audit", "io"], events)
|
||||
self.assertEqual({"discovered_by": "webdav-propfind"}, response.metadata)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
223
tests/test_connector_providers.py
Normal file
223
tests/test_connector_providers.py
Normal file
@@ -0,0 +1,223 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import patch
|
||||
|
||||
from govoplan_files.backend.storage.connector_browse import ConnectorBrowseError, _smb_location, browse_connector_profile
|
||||
from govoplan_files.backend.storage.connector_imports import ConnectorImportError, read_connector_file
|
||||
from govoplan_files.backend.storage.connector_profiles import ConnectorProfile, connector_profiles_from_payload
|
||||
from govoplan_files.backend.storage.connector_providers import connector_provider_descriptors
|
||||
|
||||
|
||||
class FakeBody:
|
||||
def __init__(self, data: bytes) -> None:
|
||||
self.data = data
|
||||
|
||||
def read(self, _limit: int) -> bytes:
|
||||
return self.data
|
||||
|
||||
|
||||
class FakeS3Client:
|
||||
def __init__(self) -> None:
|
||||
self.list_objects_request: dict[str, object] | None = None
|
||||
self.head_request: dict[str, object] | None = None
|
||||
self.get_request: dict[str, object] | None = None
|
||||
|
||||
def list_objects_v2(self, **kwargs: object) -> dict[str, object]:
|
||||
self.list_objects_request = dict(kwargs)
|
||||
return {
|
||||
"CommonPrefixes": [{"Prefix": "root/reports/"}],
|
||||
"Contents": [
|
||||
{
|
||||
"Key": "root/report.xlsx",
|
||||
"Size": 123,
|
||||
"LastModified": datetime(2026, 7, 12, 10, 0, tzinfo=UTC),
|
||||
"ETag": '"etag-1"',
|
||||
"StorageClass": "STANDARD",
|
||||
}
|
||||
],
|
||||
"NextContinuationToken": "next-page",
|
||||
}
|
||||
|
||||
def list_buckets(self) -> dict[str, object]:
|
||||
return {"Buckets": [{"Name": "archive", "CreationDate": datetime(2026, 7, 12, 9, 0, tzinfo=UTC)}]}
|
||||
|
||||
def head_object(self, **kwargs: object) -> dict[str, object]:
|
||||
self.head_request = dict(kwargs)
|
||||
return {
|
||||
"ContentLength": 13,
|
||||
"ContentType": "text/plain",
|
||||
"ETag": '"etag-2"',
|
||||
"VersionId": "version-1",
|
||||
"ChecksumSHA256": "checksum",
|
||||
}
|
||||
|
||||
def get_object(self, **kwargs: object) -> dict[str, object]:
|
||||
self.get_request = dict(kwargs)
|
||||
return {"Body": FakeBody(b"hello s3\n"), "ContentType": "text/plain", "ETag": '"etag-2"'}
|
||||
|
||||
|
||||
def s3_profile(**overrides: object) -> ConnectorProfile:
|
||||
values = {
|
||||
"id": "dev-s3",
|
||||
"label": "Dev S3",
|
||||
"provider": "s3",
|
||||
"endpoint_url": "http://127.0.0.1:9000",
|
||||
"base_path": "root",
|
||||
"credential_mode": "basic",
|
||||
"username": "access-key",
|
||||
"password_value": "secret-key",
|
||||
"metadata": {"bucket": "govoplan", "region": "eu-central-1", "path_style": True},
|
||||
}
|
||||
values.update(overrides)
|
||||
return ConnectorProfile(**values)
|
||||
|
||||
|
||||
class ConnectorProviderTests(unittest.TestCase):
|
||||
def test_smb_sdk_transport_fails_closed_before_client_creation_in_all_modes(self) -> None:
|
||||
profile = ConnectorProfile(
|
||||
id="smb",
|
||||
label="SMB",
|
||||
provider="smb",
|
||||
endpoint_url="smb://files.example.test/share",
|
||||
)
|
||||
for allow_private, address in ((False, "93.184.216.34"), (True, "10.0.0.5")):
|
||||
with self.subTest(allow_private=allow_private), patch.dict(
|
||||
"os.environ",
|
||||
{
|
||||
"APP_ENV": "production",
|
||||
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": str(allow_private).lower(),
|
||||
},
|
||||
), patch(
|
||||
"govoplan_core.security.outbound_http.socket.getaddrinfo",
|
||||
return_value=[(2, 1, 6, "", (address, 445))],
|
||||
), patch("govoplan_files.backend.storage.connector_browse._smbclient_module") as sdk, self.assertRaisesRegex(
|
||||
ConnectorBrowseError,
|
||||
"redirects/referrals.*DNS/IP pinning",
|
||||
):
|
||||
browse_connector_profile(profile, path="")
|
||||
sdk.assert_not_called()
|
||||
|
||||
def test_smb_explicit_ip_still_fails_closed_because_the_sdk_may_follow_referrals(self) -> None:
|
||||
profile = ConnectorProfile(id="smb", label="SMB", provider="smb", endpoint_url="smb://10.0.0.5/share")
|
||||
with patch.dict(
|
||||
"os.environ",
|
||||
{"APP_ENV": "production", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "true"},
|
||||
), patch(
|
||||
"govoplan_core.security.outbound_http.socket.getaddrinfo",
|
||||
return_value=[(2, 1, 6, "", ("10.0.0.5", 445))],
|
||||
), self.assertRaisesRegex(ConnectorBrowseError, "redirects/referrals.*DNS/IP pinning"):
|
||||
_smb_location(profile)
|
||||
|
||||
def test_smb_import_surfaces_fail_closed_policy_as_an_import_error(self) -> None:
|
||||
profile = ConnectorProfile(id="smb", label="SMB", provider="smb", endpoint_url="smb://10.0.0.5/share")
|
||||
with patch.dict(
|
||||
"os.environ",
|
||||
{"APP_ENV": "production", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "true"},
|
||||
), patch(
|
||||
"govoplan_core.security.outbound_http.socket.getaddrinfo",
|
||||
return_value=[(2, 1, 6, "", ("10.0.0.5", 445))],
|
||||
), patch("govoplan_files.backend.storage.connector_imports._smbclient_module") as sdk, self.assertRaisesRegex(
|
||||
ConnectorImportError,
|
||||
"redirects/referrals.*DNS/IP pinning",
|
||||
):
|
||||
read_connector_file(profile, library_id="", path="notice.txt", max_bytes=1024)
|
||||
sdk.assert_not_called()
|
||||
|
||||
def test_provider_descriptors_include_s3_and_reserved_microsoft_providers(self) -> None:
|
||||
descriptors = {descriptor.provider: descriptor for descriptor in connector_provider_descriptors()}
|
||||
|
||||
self.assertTrue(descriptors["s3"].implemented)
|
||||
self.assertTrue(descriptors["s3"].browse_supported)
|
||||
self.assertEqual("boto3", descriptors["s3"].optional_dependency)
|
||||
self.assertFalse(descriptors["sharepoint"].implemented)
|
||||
self.assertFalse(descriptors["onedrive"].implemented)
|
||||
|
||||
def test_profile_payload_accepts_new_providers_and_redacts_secret_metadata(self) -> None:
|
||||
profiles = connector_profiles_from_payload(
|
||||
{
|
||||
"profiles": [
|
||||
{
|
||||
"id": "dev-s3",
|
||||
"label": "Dev S3",
|
||||
"provider": "s3",
|
||||
"username": "access-key",
|
||||
"password": "secret-key",
|
||||
"metadata": {"bucket": "govoplan", "secret_access_key": "do-not-return"},
|
||||
},
|
||||
{"id": "sharepoint", "provider": "sharepoint"},
|
||||
{"id": "onedrive", "provider": "onedrive"},
|
||||
]
|
||||
}
|
||||
)
|
||||
|
||||
by_id = {profile.id: profile for profile in profiles}
|
||||
self.assertEqual("s3", by_id["dev-s3"].provider)
|
||||
self.assertTrue(by_id["dev-s3"].credentials_configured)
|
||||
self.assertEqual({"bucket": "govoplan"}, dict(by_id["dev-s3"].metadata))
|
||||
self.assertEqual("sharepoint", by_id["sharepoint"].provider)
|
||||
self.assertEqual("onedrive", by_id["onedrive"].provider)
|
||||
|
||||
def test_s3_browse_lists_prefixes_and_objects_with_provenance_metadata(self) -> None:
|
||||
client = FakeS3Client()
|
||||
|
||||
with patch("govoplan_files.backend.storage.connector_browse._s3_client", return_value=client):
|
||||
items = browse_connector_profile(s3_profile(), path="")
|
||||
|
||||
self.assertEqual({"Bucket": "govoplan", "Prefix": "root/", "Delimiter": "/", "MaxKeys": 1000}, client.list_objects_request)
|
||||
self.assertEqual(["folder", "file"], [item.kind for item in items])
|
||||
self.assertEqual("reports", items[0].path)
|
||||
self.assertEqual("report.xlsx", items[1].path)
|
||||
self.assertEqual("govoplan:root/report.xlsx", items[1].external_id)
|
||||
self.assertEqual("root/report.xlsx", items[1].metadata["key"])
|
||||
self.assertEqual("next-page", items[1].metadata["next_continuation_token"])
|
||||
|
||||
def test_s3_browse_lists_buckets_when_profile_has_no_bucket(self) -> None:
|
||||
client = FakeS3Client()
|
||||
|
||||
with patch("govoplan_files.backend.storage.connector_browse._s3_client", return_value=client):
|
||||
items = browse_connector_profile(s3_profile(metadata={}), path="")
|
||||
|
||||
self.assertEqual(["archive"], [item.path for item in items])
|
||||
|
||||
def test_s3_sdk_transport_fails_closed_before_client_creation_in_private_mode(self) -> None:
|
||||
with patch.dict(
|
||||
"os.environ",
|
||||
{"APP_ENV": "production", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "true"},
|
||||
), patch(
|
||||
"govoplan_core.security.outbound_http.socket.getaddrinfo",
|
||||
return_value=[(2, 1, 6, "", ("127.0.0.1", 9000))],
|
||||
), patch("govoplan_files.backend.storage.connector_browse.import_module") as importer, self.assertRaisesRegex(
|
||||
ConnectorBrowseError,
|
||||
"until that transport supports.*DNS/IP pinning",
|
||||
):
|
||||
browse_connector_profile(s3_profile(), path="")
|
||||
importer.assert_not_called()
|
||||
|
||||
def test_s3_sdk_endpoint_discovery_fails_closed(self) -> None:
|
||||
with patch("govoplan_files.backend.storage.connector_browse.import_module") as importer, self.assertRaisesRegex(
|
||||
ConnectorBrowseError,
|
||||
"endpoint discovery.*cannot guarantee.*DNS/IP pinning",
|
||||
):
|
||||
browse_connector_profile(s3_profile(endpoint_url=None), path="")
|
||||
importer.assert_not_called()
|
||||
|
||||
def test_s3_import_downloads_object_and_preserves_remote_identity(self) -> None:
|
||||
client = FakeS3Client()
|
||||
|
||||
with patch("govoplan_files.backend.storage.connector_imports._s3_client", return_value=client):
|
||||
downloaded = read_connector_file(s3_profile(), library_id="", path="report.txt", max_bytes=1024)
|
||||
|
||||
self.assertEqual({"Bucket": "govoplan", "Key": "root/report.txt"}, client.head_request)
|
||||
self.assertEqual({"Bucket": "govoplan", "Key": "root/report.txt", "VersionId": "version-1"}, client.get_request)
|
||||
self.assertEqual("report.txt", downloaded.filename)
|
||||
self.assertEqual(b"hello s3\n", downloaded.data)
|
||||
self.assertEqual("version-1", downloaded.revision)
|
||||
self.assertEqual("govoplan:root/report.txt", downloaded.external_id)
|
||||
self.assertEqual("s3://govoplan/root/report.txt", downloaded.external_url)
|
||||
self.assertEqual("checksum", downloaded.metadata["checksum_sha256"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
132
tests/test_http_client.py
Normal file
132
tests/test_http_client.py
Normal file
@@ -0,0 +1,132 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from threading import Thread
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import httpcore
|
||||
|
||||
from govoplan_core.security.outbound_http import outbound_http_policy, validate_outbound_http_url
|
||||
from govoplan_files.backend.storage.http_client import (
|
||||
ConnectorHttpError,
|
||||
_OutboundPolicyNetworkBackend,
|
||||
_SocketNetworkStream,
|
||||
request_connector_bytes,
|
||||
)
|
||||
|
||||
|
||||
class _TestHandler(BaseHTTPRequestHandler):
|
||||
def do_GET(self) -> None: # noqa: N802 - stdlib handler contract
|
||||
body = b"connector-ok"
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, _format: str, *_args: object) -> None:
|
||||
return
|
||||
|
||||
|
||||
class ConnectorHttpClientTests(unittest.TestCase):
|
||||
def test_public_transport_adapter_performs_a_real_http_request(self) -> None:
|
||||
server = ThreadingHTTPServer(("127.0.0.1", 0), _TestHandler)
|
||||
thread = Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
try:
|
||||
with patch.dict(
|
||||
"os.environ",
|
||||
{"APP_ENV": "test", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "true"},
|
||||
clear=False,
|
||||
):
|
||||
result = request_connector_bytes(
|
||||
"GET",
|
||||
f"http://127.0.0.1:{server.server_port}/object",
|
||||
)
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
thread.join(timeout=2)
|
||||
self.assertEqual(200, result.status_code)
|
||||
self.assertEqual(b"connector-ok", result.content)
|
||||
|
||||
def test_connector_responses_are_streamed_without_redirects(self) -> None:
|
||||
response = MagicMock()
|
||||
response.status_code = 200
|
||||
response.headers = {"content-length": "6"}
|
||||
response.iter_bytes.return_value = iter((b"abc", b"def"))
|
||||
context = MagicMock()
|
||||
context.__enter__.return_value = response
|
||||
|
||||
with patch(
|
||||
"govoplan_core.security.outbound_http.socket.getaddrinfo",
|
||||
return_value=[(2, 1, 6, "", ("93.184.216.34", 443))],
|
||||
), patch("govoplan_files.backend.storage.http_client._stream_connector_request", return_value=context):
|
||||
result = request_connector_bytes("GET", "https://example.test/object", max_bytes=10)
|
||||
|
||||
self.assertEqual(b"abcdef", result.content)
|
||||
|
||||
def test_connector_response_limit_is_enforced_while_streaming(self) -> None:
|
||||
response = MagicMock()
|
||||
response.status_code = 200
|
||||
response.headers = {}
|
||||
response.iter_bytes.return_value = iter((b"12345", b"67890", b"!"))
|
||||
context = MagicMock()
|
||||
context.__enter__.return_value = response
|
||||
|
||||
with patch(
|
||||
"govoplan_core.security.outbound_http.socket.getaddrinfo",
|
||||
return_value=[(2, 1, 6, "", ("93.184.216.34", 443))],
|
||||
), patch(
|
||||
"govoplan_files.backend.storage.http_client._stream_connector_request",
|
||||
return_value=context,
|
||||
), self.assertRaisesRegex(
|
||||
ConnectorHttpError,
|
||||
"configured limit",
|
||||
):
|
||||
request_connector_bytes("GET", "https://example.test/object", max_bytes=10)
|
||||
|
||||
def test_private_destination_is_blocked_before_http_connection(self) -> None:
|
||||
with patch.dict(
|
||||
"os.environ",
|
||||
{"APP_ENV": "production", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "false"},
|
||||
), patch(
|
||||
"govoplan_core.security.outbound_http.socket.getaddrinfo",
|
||||
return_value=[(2, 1, 6, "", ("10.0.0.5", 443))],
|
||||
), patch("govoplan_files.backend.storage.http_client._stream_connector_request") as stream, self.assertRaisesRegex(
|
||||
ConnectorHttpError,
|
||||
"non-public network",
|
||||
):
|
||||
request_connector_bytes("GET", "https://connector.example.test/object")
|
||||
stream.assert_not_called()
|
||||
|
||||
def test_connection_backend_rejects_private_rebinding_before_socket_open(self) -> None:
|
||||
policy = outbound_http_policy({"APP_ENV": "production"})
|
||||
public = [(2, 1, 6, "", ("93.184.216.34", 443))]
|
||||
private = [(2, 1, 6, "", ("127.0.0.1", 443))]
|
||||
with patch.dict(
|
||||
"os.environ",
|
||||
{"APP_ENV": "production", "GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "false"},
|
||||
), patch(
|
||||
"govoplan_core.security.outbound_http.socket.getaddrinfo",
|
||||
side_effect=(public, private),
|
||||
), patch("govoplan_core.security.outbound_http.socket.socket") as socket_factory:
|
||||
validate_outbound_http_url("https://connector.example.test/object", policy=policy)
|
||||
with self.assertRaises(httpcore.ConnectError):
|
||||
_OutboundPolicyNetworkBackend().connect_tcp("connector.example.test", 443)
|
||||
socket_factory.assert_not_called()
|
||||
|
||||
def test_network_backend_returns_public_network_stream_adapter(self) -> None:
|
||||
sock = MagicMock()
|
||||
with patch(
|
||||
"govoplan_files.backend.storage.http_client.create_outbound_connection",
|
||||
return_value=sock,
|
||||
):
|
||||
stream = _OutboundPolicyNetworkBackend().connect_tcp("connector.example.test", 443)
|
||||
|
||||
self.assertIsInstance(stream, _SocketNetworkStream)
|
||||
self.assertIs(stream.get_extra_info("socket"), sock)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
23
tests/test_manifest_documentation.py
Normal file
23
tests/test_manifest_documentation.py
Normal file
@@ -0,0 +1,23 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
|
||||
|
||||
class FilesManifestDocumentationTests(unittest.TestCase):
|
||||
def test_connector_topic_covers_governance_pinning_and_provenance_perspectives(self) -> None:
|
||||
from govoplan_files.backend.manifest import manifest
|
||||
|
||||
topic = next(item for item in manifest.documentation if item.id == "files.governed-connectors-and-provenance")
|
||||
|
||||
self.assertEqual(("admin", "user"), topic.documentation_types)
|
||||
self.assertEqual({"user", "admin", "operator"}, set(topic.metadata["perspectives"]))
|
||||
self.assertIn("fails closed", topic.body)
|
||||
self.assertIn("DFS referrals", topic.body)
|
||||
self.assertIn("immediately scrubs", topic.body)
|
||||
self.assertTrue(any("non-owned external references" in item for item in topic.metadata["security_invariants"]))
|
||||
self.assertIn("revision", topic.metadata["provenance_fields"])
|
||||
self.assertIn("/api/v1/files/connectors/profiles", {link.href for link in topic.links})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
36
tests/test_router_contract.py
Normal file
36
tests/test_router_contract.py
Normal file
@@ -0,0 +1,36 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
|
||||
from govoplan_files.backend.router import router
|
||||
|
||||
|
||||
class FilesRouterContractTests(unittest.TestCase):
|
||||
def test_connector_routes_keep_existing_api_paths(self) -> None:
|
||||
routes = {(tuple(sorted(route.methods or ())), route.path) for route in router.routes}
|
||||
|
||||
expected = {
|
||||
(("GET",), "/files/connectors/providers"),
|
||||
(("GET",), "/files/connectors/settings/delta"),
|
||||
(("GET",), "/files/connectors/profiles"),
|
||||
(("POST",), "/files/connectors/profiles"),
|
||||
(("GET",), "/files/connectors/profiles/{profile_id}/browse"),
|
||||
(("POST",), "/files/connectors/profiles/{profile_id}/import"),
|
||||
(("POST",), "/files/connectors/profiles/{profile_id}/sync"),
|
||||
(("GET",), "/files/connectors/credentials"),
|
||||
(("POST",), "/files/connectors/credentials"),
|
||||
(("GET",), "/files/connector-spaces"),
|
||||
(("POST",), "/files/connector-spaces"),
|
||||
}
|
||||
|
||||
self.assertTrue(expected.issubset(routes))
|
||||
|
||||
def test_bulk_organize_routes_keep_existing_api_paths(self) -> None:
|
||||
routes = {(tuple(sorted(route.methods or ())), route.path) for route in router.routes}
|
||||
|
||||
self.assertIn((("POST",), "/files/bulk-rename"), routes)
|
||||
self.assertIn((("POST",), "/files/transfer"), routes)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
78
tests/test_storage_backends.py
Normal file
78
tests/test_storage_backends.py
Normal file
@@ -0,0 +1,78 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, PropertyMock, patch
|
||||
|
||||
from govoplan_files.backend.storage.backends import S3StorageBackend, StorageBackendError
|
||||
|
||||
|
||||
def _backend() -> S3StorageBackend:
|
||||
return S3StorageBackend(
|
||||
bucket="files",
|
||||
endpoint_url="https://objects.example.test",
|
||||
region_name="test",
|
||||
access_key_id="access",
|
||||
secret_access_key="secret",
|
||||
)
|
||||
|
||||
|
||||
class S3StorageBackendTests(unittest.TestCase):
|
||||
def test_sdk_transport_fails_closed_in_public_and_private_modes(self) -> None:
|
||||
for allow_private, address in ((False, "93.184.216.34"), (True, "10.0.0.5")):
|
||||
with self.subTest(allow_private=allow_private), patch.dict(
|
||||
"os.environ",
|
||||
{
|
||||
"APP_ENV": "production",
|
||||
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": str(allow_private).lower(),
|
||||
},
|
||||
), patch(
|
||||
"govoplan_core.security.outbound_http.socket.getaddrinfo",
|
||||
return_value=[(2, 1, 6, "", (address, 443))],
|
||||
), self.assertRaisesRegex(StorageBackendError, "until that transport supports.*DNS/IP pinning"):
|
||||
_backend().client
|
||||
|
||||
def test_get_bytes_rejects_declared_oversize_object_without_reading(self) -> None:
|
||||
body = MagicMock()
|
||||
client = MagicMock()
|
||||
client.get_object.return_value = {"ContentLength": 6, "Body": body}
|
||||
with patch.dict("os.environ", {"GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES": "5"}), patch.object(
|
||||
S3StorageBackend,
|
||||
"client",
|
||||
new_callable=PropertyMock,
|
||||
return_value=client,
|
||||
), self.assertRaisesRegex(StorageBackendError, "deployment limit"):
|
||||
_backend().get_bytes("large.bin")
|
||||
body.read.assert_not_called()
|
||||
body.close.assert_called_once_with()
|
||||
|
||||
def test_iter_bytes_rejects_undeclared_oversize_object(self) -> None:
|
||||
client = MagicMock()
|
||||
client.get_object.return_value = {"Body": io.BytesIO(b"123456")}
|
||||
with patch.dict("os.environ", {"GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES": "5"}), patch.object(
|
||||
S3StorageBackend,
|
||||
"client",
|
||||
new_callable=PropertyMock,
|
||||
return_value=client,
|
||||
), self.assertRaisesRegex(StorageBackendError, "deployment limit"):
|
||||
list(_backend().iter_bytes("large.bin", chunk_size=3))
|
||||
|
||||
def test_iter_bytes_closes_streaming_body_when_consumer_stops_early(self) -> None:
|
||||
body = MagicMock()
|
||||
body.read.side_effect = (b"123", b"456", b"")
|
||||
client = MagicMock()
|
||||
client.get_object.return_value = {"ContentLength": 6, "Body": body}
|
||||
with patch.dict("os.environ", {"GOVOPLAN_CONNECTOR_MAX_FILE_TRANSFER_BYTES": "10"}), patch.object(
|
||||
S3StorageBackend,
|
||||
"client",
|
||||
new_callable=PropertyMock,
|
||||
return_value=client,
|
||||
):
|
||||
chunks = _backend().iter_bytes("object.bin", chunk_size=3)
|
||||
self.assertEqual(b"123", next(chunks))
|
||||
chunks.close()
|
||||
body.close.assert_called_once_with()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
67
tests/test_transfer_helpers.py
Normal file
67
tests/test_transfer_helpers.py
Normal file
@@ -0,0 +1,67 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
|
||||
from govoplan_files.backend.storage.common import FileConflictResolution, FileStorageError
|
||||
from govoplan_files.backend.storage.transfers import _normalize_rename_request, _normalize_transfer_request
|
||||
|
||||
|
||||
class TransferHelperTests(unittest.TestCase):
|
||||
def test_transfer_normalization_deduplicates_folder_paths_and_conflicts(self) -> None:
|
||||
normalized = _normalize_transfer_request(
|
||||
operation=" COPY ",
|
||||
source_owner_type=" USER ",
|
||||
target_owner_type=" GROUP ",
|
||||
folder_paths=["Reports", "Reports/2026", "", "./Archive"],
|
||||
target_folder=" Target ",
|
||||
conflict_strategy="rename",
|
||||
conflict_resolutions=[FileConflictResolution(target_path="Target/a.txt", action="skip")],
|
||||
)
|
||||
|
||||
self.assertEqual("copy", normalized.operation)
|
||||
self.assertEqual("user", normalized.source_owner_type)
|
||||
self.assertEqual("group", normalized.target_owner_type)
|
||||
self.assertEqual(["Reports", "Reports/2026", "Archive"], normalized.source_folder_paths)
|
||||
self.assertEqual("Target", normalized.target_folder)
|
||||
self.assertEqual("rename", normalized.conflict_strategy)
|
||||
self.assertEqual("skip", normalized.conflict_resolution_map["Target/a.txt"].action)
|
||||
|
||||
def test_transfer_normalization_rejects_unknown_operation(self) -> None:
|
||||
with self.assertRaisesRegex(FileStorageError, "Unsupported transfer operation"):
|
||||
_normalize_transfer_request(
|
||||
operation="link",
|
||||
source_owner_type="user",
|
||||
target_owner_type="group",
|
||||
folder_paths=[],
|
||||
target_folder="",
|
||||
conflict_strategy="reject",
|
||||
conflict_resolutions=None,
|
||||
)
|
||||
|
||||
def test_rename_normalization_collapses_nested_folder_roots(self) -> None:
|
||||
normalized = _normalize_rename_request(
|
||||
file_ids=["file-1", "file-1", "file-2"],
|
||||
folder_paths=["Reports", "Reports/2026", "Archive"],
|
||||
owner_type=" USER ",
|
||||
owner_id="owner-1",
|
||||
mode=" prefix ",
|
||||
)
|
||||
|
||||
self.assertEqual("prefix", normalized.mode)
|
||||
self.assertEqual(["file-1", "file-2"], normalized.selected_file_ids)
|
||||
self.assertEqual(["Archive", "Reports"], normalized.selected_folder_roots)
|
||||
self.assertEqual("user", normalized.owner_type)
|
||||
|
||||
def test_rename_normalization_rejects_invalid_direct_multi_select(self) -> None:
|
||||
with self.assertRaisesRegex(FileStorageError, "Direct rename requires exactly one selected item"):
|
||||
_normalize_rename_request(
|
||||
file_ids=["file-1", "file-2"],
|
||||
folder_paths=[],
|
||||
owner_type="user",
|
||||
owner_id="owner-1",
|
||||
mode="direct",
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@govoplan/files-webui",
|
||||
"version": "0.1.8",
|
||||
"version": "0.1.9",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
@@ -13,6 +13,9 @@
|
||||
},
|
||||
"./styles/file-manager.css": "./src/styles/file-manager.css"
|
||||
},
|
||||
"scripts": {
|
||||
"test:file-drop-target": "node scripts/test-file-drop-target-structure.mjs"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@vitejs/plugin-react": "^4.3.4",
|
||||
"vite": "^6.0.6",
|
||||
@@ -21,7 +24,7 @@
|
||||
"react-dom": "^19.0.0",
|
||||
"react-router-dom": "^7.1.1",
|
||||
"lucide-react": "^1.23.0",
|
||||
"@govoplan/core-webui": "^0.1.8"
|
||||
"@govoplan/core-webui": "^0.1.9"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@govoplan/core-webui": {
|
||||
|
||||
35
webui/scripts/test-file-drop-target-structure.mjs
Normal file
35
webui/scripts/test-file-drop-target-structure.mjs
Normal file
@@ -0,0 +1,35 @@
|
||||
import { readFileSync } from "node:fs";
|
||||
|
||||
const source = readFileSync(new URL("../src/features/files/FilesPage.tsx", import.meta.url), "utf8");
|
||||
const normalized = source.replace(/\s+/g, " ");
|
||||
|
||||
function assertIncludes(fragment, message) {
|
||||
if (!normalized.includes(fragment.replace(/\s+/g, " "))) throw new Error(message);
|
||||
}
|
||||
|
||||
assertIncludes(
|
||||
"const target = options.target ?? currentActionTarget();",
|
||||
"uploads must prefer the explicit target supplied by the initiating interaction"
|
||||
);
|
||||
assertIncludes(
|
||||
"const targetSpace = target ? findSpace(target.spaceId) : null;",
|
||||
"the upload owner must be resolved from the selected target"
|
||||
);
|
||||
assertIncludes(
|
||||
"owner_type: targetSpace.owner_type, owner_id: targetSpace.owner_id, path: target.folderPath,",
|
||||
"the upload request must use the selected target owner and folder"
|
||||
);
|
||||
assertIncludes(
|
||||
"async function uploadExternalFilesToTarget(fileList: FileList | File[], target: FileActionTarget) { await handleFilesUpload(fileList, { target }); }",
|
||||
"external drops must pass their concrete target without asynchronous dialog-state mutation"
|
||||
);
|
||||
assertIncludes(
|
||||
"await uploadExternalFilesToTarget(event.dataTransfer.files, target);",
|
||||
"drop handling must forward the target on which the files were dropped"
|
||||
);
|
||||
assertIncludes(
|
||||
"onFiles={(files) => handleFilesUpload(files, { target: activeDialogTarget || undefined })}",
|
||||
"the dialog picker/drop zone must snapshot its selected target for upload"
|
||||
);
|
||||
|
||||
console.log("Files upload target routing structure is intact.");
|
||||
@@ -1,4 +1,10 @@
|
||||
import { ApiError, apiFetch, apiUrl, authHeaders, csrfToken, type ApiSettings, type DeltaDeletedItem, type FilesManagedFileLinkTarget } from "@govoplan/core-webui";
|
||||
export { fetchResourceAccessExplanation } from "@govoplan/core-webui";
|
||||
export type {
|
||||
AccessDecisionProvenanceItem,
|
||||
ResourceAccessExplanationUser as AccessExplanationUser,
|
||||
ResourceAccessExplanationResponse
|
||||
} from "@govoplan/core-webui";
|
||||
|
||||
export type FileSpace = {
|
||||
id: string;
|
||||
@@ -116,8 +122,6 @@ export type FileConnectorCredentialsPayload = {
|
||||
username?: string | null;
|
||||
password?: string | null;
|
||||
token?: string | null;
|
||||
password_env?: string | null;
|
||||
token_env?: string | null;
|
||||
secret_ref?: string | null;
|
||||
};
|
||||
|
||||
@@ -150,7 +154,7 @@ export type FileConnectorDiscoveryResponse = {
|
||||
export type FileConnectorProfilePayload = {
|
||||
id: string;
|
||||
label: string;
|
||||
provider: "seafile" | "nextcloud" | "webdav" | "smb" | "nfs" | "dms" | "generic";
|
||||
provider: "seafile" | "nextcloud" | "webdav" | "smb" | "s3" | "sharepoint" | "onedrive" | "nfs" | "dms" | "generic";
|
||||
endpoint_url?: string | null;
|
||||
base_path?: string | null;
|
||||
enabled?: boolean;
|
||||
@@ -172,7 +176,7 @@ export type FileConnectorProfileUpdatePayload = Partial<Omit<FileConnectorProfil
|
||||
export type FileConnectorCredentialPayload = {
|
||||
id: string;
|
||||
label: string;
|
||||
provider?: "seafile" | "nextcloud" | "webdav" | "smb" | "nfs" | "dms" | "generic" | null;
|
||||
provider?: "seafile" | "nextcloud" | "webdav" | "smb" | "s3" | "sharepoint" | "onedrive" | "nfs" | "dms" | "generic" | null;
|
||||
enabled?: boolean;
|
||||
scope_type?: "system" | "tenant" | "user" | "group" | "campaign";
|
||||
scope_id?: string | null;
|
||||
@@ -206,6 +210,8 @@ export type FileConnectorBrowseResponse = {
|
||||
path: string;
|
||||
library_id?: string | null;
|
||||
read_only: boolean;
|
||||
next_continuation_token?: string | null;
|
||||
has_more: boolean;
|
||||
decision: FileConnectorPolicyDecision;
|
||||
items: FileConnectorBrowseItem[];
|
||||
};
|
||||
@@ -559,44 +565,6 @@ export function bulkDeleteFiles(settings: ApiSettings, fileIds: string[]): Promi
|
||||
|
||||
export type FileBulkShareResponse = {shares: FileShare[];shared_count: number;};
|
||||
|
||||
export type AccessExplanationUser = {
|
||||
id: string;
|
||||
account_id?: string | null;
|
||||
email?: string | null;
|
||||
display_name?: string | null;
|
||||
};
|
||||
|
||||
export type AccessDecisionProvenanceItem = {
|
||||
kind: string;
|
||||
id?: string | null;
|
||||
label?: string | null;
|
||||
tenant_id?: string | null;
|
||||
source?: string | null;
|
||||
details?: Record<string, unknown>;
|
||||
};
|
||||
|
||||
export type ResourceAccessExplanationResponse = {
|
||||
user: AccessExplanationUser;
|
||||
resource_type: string;
|
||||
resource_id: string;
|
||||
action: string;
|
||||
provenance: AccessDecisionProvenanceItem[];
|
||||
};
|
||||
|
||||
export function fetchResourceAccessExplanation(
|
||||
settings: ApiSettings,
|
||||
options: {userId: string;resourceType: string;resourceId: string;action: string;tenantId?: string | null;})
|
||||
: Promise<ResourceAccessExplanationResponse> {
|
||||
const params = new URLSearchParams({
|
||||
user_id: options.userId,
|
||||
resource_type: options.resourceType,
|
||||
resource_id: options.resourceId,
|
||||
action: options.action
|
||||
});
|
||||
if (options.tenantId) params.set("tenant_id", options.tenantId);
|
||||
return apiFetch<ResourceAccessExplanationResponse>(settings, `/api/v1/admin/access/resource-explanation?${params.toString()}`);
|
||||
}
|
||||
|
||||
export function virtualFolderResourceId(options: {tenantId: string;ownerType: "user" | "group";ownerId: string;path: string;}): string {
|
||||
return `virtual-folder:v1:${options.tenantId}:${options.ownerType}:${options.ownerId}:${base64Url(options.path.replace(/\\/g, "/").replace(/^\/+|\/+$/g, ""))}`;
|
||||
}
|
||||
@@ -808,11 +776,12 @@ export function deactivateFileConnectorProfile(settings: ApiSettings, profileId:
|
||||
export function browseFileConnectorProfile(
|
||||
settings: ApiSettings,
|
||||
profileId: string,
|
||||
params: {path?: string;library_id?: string;campaign_id?: string;} = {})
|
||||
params: {path?: string;library_id?: string;continuation_token?: string;campaign_id?: string;} = {})
|
||||
: Promise<FileConnectorBrowseResponse> {
|
||||
const search = new URLSearchParams();
|
||||
if (params.path) search.set("path", params.path);
|
||||
if (params.library_id) search.set("library_id", params.library_id);
|
||||
if (params.continuation_token) search.set("continuation_token", params.continuation_token);
|
||||
if (params.campaign_id) search.set("campaign_id", params.campaign_id);
|
||||
const suffix = search.toString() ? `?${search.toString()}` : "";
|
||||
return apiFetch<FileConnectorBrowseResponse>(settings, `/api/v1/files/connectors/profiles/${encodeURIComponent(profileId)}/browse${suffix}`);
|
||||
|
||||
@@ -5,6 +5,8 @@ import {
|
||||
Card,
|
||||
ConnectionTree,
|
||||
ConfirmDialog,
|
||||
CredentialPanel,
|
||||
DisabledActionTooltip,
|
||||
DismissibleAlert,
|
||||
Dialog,
|
||||
FormField,
|
||||
@@ -13,6 +15,8 @@ import {
|
||||
LoadingFrame,
|
||||
mergeDeltaRows,
|
||||
SegmentedControl,
|
||||
StatusBadge,
|
||||
TableActionGroup,
|
||||
ToggleSwitch,
|
||||
useDeltaWatermarks,
|
||||
useUnsavedDraftGuard,
|
||||
@@ -63,8 +67,6 @@ type ConnectorProfileDraft = {
|
||||
username: string;
|
||||
password: string;
|
||||
token: string;
|
||||
passwordEnv: string;
|
||||
tokenEnv: string;
|
||||
secretRef: string;
|
||||
canBrowse: boolean;
|
||||
canImport: boolean;
|
||||
@@ -89,8 +91,6 @@ type ConnectorCredentialDraft = {
|
||||
username: string;
|
||||
password: string;
|
||||
token: string;
|
||||
passwordEnv: string;
|
||||
tokenEnv: string;
|
||||
secretRef: string;
|
||||
policyMode: PolicyMode;
|
||||
allowedPaths: string;
|
||||
@@ -123,6 +123,9 @@ const PROVIDERS: Array<{id: Provider;label: string;}> = [
|
||||
{ id: "nextcloud", label: "i18n:govoplan-files.nextcloud.aab73a3d" },
|
||||
{ id: "seafile", label: "i18n:govoplan-files.seafile.600192cc" },
|
||||
{ id: "smb", label: "i18n:govoplan-files.smb.515d2f88" },
|
||||
{ id: "s3", label: "S3" },
|
||||
{ id: "sharepoint", label: "SharePoint" },
|
||||
{ id: "onedrive", label: "OneDrive" },
|
||||
{ id: "nfs", label: "i18n:govoplan-files.nfs.db121865" },
|
||||
{ id: "dms", label: "i18n:govoplan-files.dms.477e5652" },
|
||||
{ id: "generic", label: "i18n:govoplan-files.generic.ff7613e5" }];
|
||||
@@ -133,6 +136,9 @@ const ENDPOINT_PLACEHOLDERS: Record<Provider, string> = {
|
||||
nextcloud: "http://127.0.0.1:9081/remote.php/dav/files/admin/",
|
||||
seafile: "http://127.0.0.1:9082/",
|
||||
smb: "smb://127.0.0.1:1445/files",
|
||||
s3: "http://127.0.0.1:9000",
|
||||
sharepoint: "https://graph.microsoft.com/v1.0/sites/{site-id}/drives/{drive-id}",
|
||||
onedrive: "https://graph.microsoft.com/v1.0/drives/{drive-id}",
|
||||
nfs: "nfs://127.0.0.1/export",
|
||||
dms: "https://dms.example.test",
|
||||
generic: "https://files.example.test"
|
||||
@@ -340,6 +346,17 @@ export default function FileConnectorSettingsPanel({
|
||||
setCredentialDraft((current) => current ? { ...current, ...patch } : current);
|
||||
}
|
||||
|
||||
function patchCredentialValues(patch: {username?: string | null;password?: string | null;}) {
|
||||
const next: Partial<ConnectorCredentialDraft> = {};
|
||||
if (patch.username !== undefined) next.username = String(patch.username ?? "");
|
||||
if (patch.password !== undefined) next.password = String(patch.password ?? "");
|
||||
patchCredentialDraft(next);
|
||||
}
|
||||
|
||||
function patchCredentialToken(patch: {password?: string | null;}) {
|
||||
if (patch.password !== undefined) patchCredentialDraft({ token: String(patch.password ?? "") });
|
||||
}
|
||||
|
||||
function patchPolicyDraft(patch: Partial<CentralConnectorPolicyDraft>) {
|
||||
setPolicyDraft((current) => ({ ...current, ...patch }));
|
||||
}
|
||||
@@ -472,8 +489,6 @@ export default function FileConnectorSettingsPanel({
|
||||
username: cleanOrNull(credentialDraft.username),
|
||||
password: cleanOrUndefined(credentialDraft.password),
|
||||
token: cleanOrUndefined(credentialDraft.token),
|
||||
password_env: cleanOrNull(credentialDraft.passwordEnv),
|
||||
token_env: cleanOrNull(credentialDraft.tokenEnv),
|
||||
secret_ref: cleanOrNull(credentialDraft.secretRef)
|
||||
};
|
||||
const sharedPayload = {
|
||||
@@ -614,8 +629,6 @@ export default function FileConnectorSettingsPanel({
|
||||
username: cleanOrNull(draft.username),
|
||||
password: cleanOrUndefined(draft.password),
|
||||
token: cleanOrUndefined(draft.token),
|
||||
password_env: cleanOrNull(draft.passwordEnv),
|
||||
token_env: cleanOrNull(draft.tokenEnv),
|
||||
secret_ref: cleanOrNull(draft.secretRef)
|
||||
},
|
||||
metadata: metadataFromDraft(draft)
|
||||
@@ -661,8 +674,6 @@ export default function FileConnectorSettingsPanel({
|
||||
username: cleanOrNull(credentialDraft.username),
|
||||
password: cleanOrUndefined(credentialDraft.password),
|
||||
token: cleanOrUndefined(credentialDraft.token),
|
||||
password_env: cleanOrNull(credentialDraft.passwordEnv),
|
||||
token_env: cleanOrNull(credentialDraft.tokenEnv),
|
||||
secret_ref: cleanOrNull(credentialDraft.secretRef)
|
||||
},
|
||||
metadata: profile.metadata ?? {},
|
||||
@@ -737,7 +748,7 @@ export default function FileConnectorSettingsPanel({
|
||||
width: "120px",
|
||||
render: (row) => {
|
||||
const enabled = row.kind === "profile" ? row.profile.enabled : row.credential.enabled;
|
||||
return <span className={`status-badge ${enabled ? "success" : "neutral"}`}>{enabled ? "i18n:govoplan-files.enabled.df174a3f" : "i18n:govoplan-files.disabled.f4f4473d"}</span>;
|
||||
return <StatusBadge status={enabled ? "success" : "inactive"} label={enabled ? "i18n:govoplan-files.enabled.df174a3f" : "i18n:govoplan-files.disabled.f4f4473d"} />;
|
||||
}
|
||||
}];
|
||||
|
||||
@@ -752,21 +763,19 @@ export default function FileConnectorSettingsPanel({
|
||||
function renderConnectorActions(row: ConnectorTreeRow) {
|
||||
if (row.kind === "credential") {
|
||||
const readOnly = row.credential.source_kind !== "database";
|
||||
return (
|
||||
<div className="admin-icon-actions">
|
||||
<Button type="button" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.credential.label })} aria-label={i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.credential.label })} onClick={() => startCredentialEdit(row.credential)} disabled={!canWrite || readOnly || saving}><Edit3 size={15} /></Button>
|
||||
<Button type="button" variant="danger" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.credential.label })} aria-label={i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.credential.label })} onClick={() => setPendingCredentialDeactivate(row.credential)} disabled={!canWrite || readOnly || saving || !row.credential.enabled}><Trash2 size={15} /></Button>
|
||||
</div>);
|
||||
return <TableActionGroup actions={[
|
||||
{ id: "edit", label: i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.credential.label }), icon: <Edit3 size={15} />, disabled: !canWrite || readOnly || saving, onClick: () => startCredentialEdit(row.credential) },
|
||||
{ id: "disable", label: i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.credential.label }), icon: <Trash2 size={15} />, variant: "danger", applicable: row.credential.enabled, disabled: !canWrite || readOnly || saving, onClick: () => setPendingCredentialDeactivate(row.credential) }
|
||||
]} />;
|
||||
|
||||
}
|
||||
const readOnly = row.profile.source_kind !== "database";
|
||||
return (
|
||||
<div className="admin-icon-actions">
|
||||
<Button type="button" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.test_value.6a5d10a5", { value0: row.profile.label })} aria-label={i18nMessage("i18n:govoplan-files.test_value.6a5d10a5", { value0: row.profile.label })} onClick={() => void testBrowse(row.profile)} disabled={saving || testingProfileId === row.profile.id || !row.profile.enabled}><RefreshCw size={15} /></Button>
|
||||
<Button type="button" variant="primary" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.add_credential_for_value.0fa9c1fe", { value0: row.profile.label })} aria-label={i18nMessage("i18n:govoplan-files.add_credential_for_value.0fa9c1fe", { value0: row.profile.label })} onClick={() => startCredentialCreate(row.profile)} disabled={!canWrite || saving}><UserRoundKey size={15} /></Button>
|
||||
<Button type="button" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.profile.label })} aria-label={i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.profile.label })} onClick={() => startEdit(row.profile)} disabled={!canWrite || readOnly || saving}><Edit3 size={15} /></Button>
|
||||
<Button type="button" variant="danger" className="admin-icon-button" title={i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.profile.label })} aria-label={i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.profile.label })} onClick={() => setPendingDeactivate(row.profile)} disabled={!canWrite || readOnly || saving || !row.profile.enabled}><Trash2 size={15} /></Button>
|
||||
</div>);
|
||||
return <TableActionGroup actions={[
|
||||
{ id: "test", label: i18nMessage("i18n:govoplan-files.test_value.6a5d10a5", { value0: row.profile.label }), icon: <RefreshCw size={15} />, applicable: row.profile.enabled, disabled: saving || testingProfileId === row.profile.id, onClick: () => void testBrowse(row.profile) },
|
||||
{ id: "add-credential", label: i18nMessage("i18n:govoplan-files.add_credential_for_value.0fa9c1fe", { value0: row.profile.label }), icon: <UserRoundKey size={15} />, variant: "primary", disabled: !canWrite || saving, onClick: () => startCredentialCreate(row.profile) },
|
||||
{ id: "edit", label: i18nMessage("i18n:govoplan-files.edit_value.fad75899", { value0: row.profile.label }), icon: <Edit3 size={15} />, disabled: !canWrite || readOnly || saving, onClick: () => startEdit(row.profile) },
|
||||
{ id: "disable", label: i18nMessage("i18n:govoplan-files.disable_value.09485f7f", { value0: row.profile.label }), icon: <Trash2 size={15} />, variant: "danger", applicable: row.profile.enabled, disabled: !canWrite || readOnly || saving, onClick: () => setPendingDeactivate(row.profile) }
|
||||
]} />;
|
||||
|
||||
}
|
||||
|
||||
@@ -801,7 +810,7 @@ export default function FileConnectorSettingsPanel({
|
||||
|
||||
{targetSelectionRequired &&
|
||||
<Card title={i18nMessage("i18n:govoplan-files.value_scope", { value0: targetLabel })}>
|
||||
<div className="mail-profile-target-row">
|
||||
<div className="settings-target-row">
|
||||
<FormField label={targetLabel}>
|
||||
<select value={selectedTargetId} onChange={(event) => setSelectedTargetId(event.target.value)} disabled={loading || saving || !hasSelectableTarget}>
|
||||
{!hasSelectableTarget && <option value="">{targetEmptyText}</option>}
|
||||
@@ -846,7 +855,7 @@ export default function FileConnectorSettingsPanel({
|
||||
|
||||
<LoadingFrame loading={loading} label="i18n:govoplan-files.loading_connector_policy.f12ab285">
|
||||
<>
|
||||
<div className="form-grid two-column-form-grid">
|
||||
<div className="form-grid two">
|
||||
<FormField label="i18n:govoplan-files.allowed_connection_ids.0df854c8">
|
||||
<textarea value={policyDraft.allowedConnectors} disabled={saving || !canWrite} onChange={(event) => patchPolicyDraft({ allowedConnectors: event.target.value })} rows={3} placeholder={"tenant-webdav\nseafile-*"} />
|
||||
</FormField>
|
||||
@@ -906,11 +915,11 @@ export default function FileConnectorSettingsPanel({
|
||||
footer={credentialDraft ?
|
||||
<>
|
||||
<Button onClick={closeCredentialDialog} disabled={saving}>i18n:govoplan-files.cancel.77dfd213</Button>
|
||||
<span className="disabled-action-tooltip" data-tooltip={credentialSaveTooltip}>
|
||||
<DisabledActionTooltip reason={credentialSaveTooltip}>
|
||||
<Button variant="primary" onClick={() => void saveCredentialDraft()} disabled={credentialSaveDisabled}>
|
||||
<Save size={16} aria-hidden="true" /> {saving ? "i18n:govoplan-files.saving.ae7e8875" : "i18n:govoplan-files.save_credential.2c02a6d2"}
|
||||
</Button>
|
||||
</span>
|
||||
</DisabledActionTooltip>
|
||||
</> :
|
||||
null}>
|
||||
|
||||
@@ -921,7 +930,7 @@ export default function FileConnectorSettingsPanel({
|
||||
<h3>Credential</h3>
|
||||
<p>Choose where this credential can be used and how it signs in.</p>
|
||||
</header>
|
||||
<div className="form-grid two-column-form-grid">
|
||||
<div className="form-grid two">
|
||||
<FormField label="i18n:govoplan-files.credential_id.9432a6e1">
|
||||
<input className={!editingCredentialId && !credentialDraft.id.trim() ? "field-input-missing" : undefined} aria-invalid={!editingCredentialId && !credentialDraft.id.trim() || undefined} value={credentialDraft.id} disabled={Boolean(editingCredentialId) || saving} onChange={(event) => patchCredentialDraft({ id: event.target.value })} placeholder={`${scopeType}-webdav-credentials`} />
|
||||
</FormField>
|
||||
@@ -983,51 +992,53 @@ export default function FileConnectorSettingsPanel({
|
||||
actor: "Connector administrator",
|
||||
target: "Credential mode"
|
||||
}} /> :
|
||||
<div className="form-grid two-column-form-grid">
|
||||
{(credentialDraft.credentialMode === "basic" || credentialDraft.credentialMode === "secret_ref") &&
|
||||
<FormField label="i18n:govoplan-files.username.84c29015">
|
||||
<input value={credentialDraft.username} disabled={saving} onChange={(event) => patchCredentialDraft({ username: event.target.value })} autoComplete="username" />
|
||||
</FormField>
|
||||
}
|
||||
<div className="file-connector-secret-fields">
|
||||
{credentialDraft.credentialMode === "basic" &&
|
||||
<FormField label="i18n:govoplan-files.password.8be3c943">
|
||||
<input value={credentialDraft.password} disabled={saving} onChange={(event) => patchCredentialDraft({ password: event.target.value })} type="password" autoComplete="new-password" placeholder={editingCredentialId ? "i18n:govoplan-files.leave_empty_to_keep_saved_password.6ec39f5e" : ""} />
|
||||
</FormField>
|
||||
}
|
||||
{credentialDraft.credentialMode === "token" &&
|
||||
<FormField label="i18n:govoplan-files.token.a1141eb9">
|
||||
<input value={credentialDraft.token} disabled={saving} onChange={(event) => patchCredentialDraft({ token: event.target.value })} type="password" placeholder={editingCredentialId ? "i18n:govoplan-files.leave_empty_to_keep_saved_token.e725857b" : ""} />
|
||||
</FormField>
|
||||
<CredentialPanel
|
||||
values={{ username: credentialDraft.username, password: credentialDraft.password }}
|
||||
onChange={patchCredentialValues}
|
||||
disabled={saving}
|
||||
savedPassword={Boolean(editingCredentialId) && !credentialDraft.clearPassword}
|
||||
savedPasswordPlaceholder="i18n:govoplan-files.leave_empty_to_keep_saved_password.6ec39f5e" />
|
||||
}
|
||||
{credentialDraft.credentialMode === "secret_ref" &&
|
||||
<>
|
||||
<CredentialPanel
|
||||
values={{ username: credentialDraft.username }}
|
||||
onChange={patchCredentialValues}
|
||||
disabled={saving}
|
||||
showPassword={false} />
|
||||
<div className="form-grid two">
|
||||
<FormField label="i18n:govoplan-files.secret_reference.04ed2221">
|
||||
<input value={credentialDraft.secretRef} disabled={saving} onChange={(event) => patchCredentialDraft({ secretRef: event.target.value })} />
|
||||
</FormField>
|
||||
</div>
|
||||
</>
|
||||
}
|
||||
{credentialDraft.credentialMode === "token" &&
|
||||
<CredentialPanel
|
||||
values={{ password: credentialDraft.token }}
|
||||
onChange={patchCredentialToken}
|
||||
disabled={saving}
|
||||
showUsername={false}
|
||||
passwordLabel="i18n:govoplan-files.token.a1141eb9"
|
||||
savedPassword={Boolean(editingCredentialId) && !credentialDraft.clearToken}
|
||||
savedPasswordPlaceholder="i18n:govoplan-files.leave_empty_to_keep_saved_token.e725857b" />
|
||||
}
|
||||
</div>
|
||||
}
|
||||
<div className="button-row compact-actions">
|
||||
<span className="disabled-action-tooltip" data-tooltip={credentialTestDisabled ? credentialTestTooltip : ""}>
|
||||
<DisabledActionTooltip reason={credentialTestDisabled ? credentialTestTooltip : ""}>
|
||||
<Button type="button" onClick={() => void testCredentialLogin()} disabled={credentialTestDisabled}>
|
||||
<RefreshCw size={16} aria-hidden="true" /> {testingCredentialLogin ? "Testing login" : "Test login"}
|
||||
</Button>
|
||||
</span>
|
||||
</DisabledActionTooltip>
|
||||
</div>
|
||||
{credentialLoginResult &&
|
||||
<DismissibleAlert tone={credentialLoginResult.ok ? "success" : "warning"} resetKey={credentialLoginResult.message}>
|
||||
{credentialLoginResult.message}
|
||||
</DismissibleAlert>
|
||||
}
|
||||
<AdvancedOptionsPanel title="Environment and fallback secrets" summary="Use these only when the deployment injects secrets outside the database.">
|
||||
<div className="form-grid two-column-form-grid">
|
||||
<FormField label="i18n:govoplan-files.password_environment_variable.0e77673f">
|
||||
<input value={credentialDraft.passwordEnv} disabled={saving} onChange={(event) => patchCredentialDraft({ passwordEnv: event.target.value })} />
|
||||
</FormField>
|
||||
<FormField label="i18n:govoplan-files.token_environment_variable.5af4a79e">
|
||||
<input value={credentialDraft.tokenEnv} disabled={saving} onChange={(event) => patchCredentialDraft({ tokenEnv: event.target.value })} />
|
||||
</FormField>
|
||||
</div>
|
||||
</AdvancedOptionsPanel>
|
||||
</section>
|
||||
|
||||
<section className="adaptive-config-section">
|
||||
@@ -1035,7 +1046,7 @@ export default function FileConnectorSettingsPanel({
|
||||
<h3>Policy</h3>
|
||||
<p>Limit where lower levels may use this credential.</p>
|
||||
</header>
|
||||
<div className="form-grid two-column-form-grid">
|
||||
<div className="form-grid two">
|
||||
<FormField label="i18n:govoplan-files.policy.bb9cf141">
|
||||
<select value={credentialDraft.policyMode} disabled={saving} onChange={(event) => patchCredentialDraft({ policyMode: event.target.value as PolicyMode })}>
|
||||
<option value="allow-provider">i18n:govoplan-files.can_use_this_credential.f4a41971</option>
|
||||
@@ -1072,11 +1083,11 @@ export default function FileConnectorSettingsPanel({
|
||||
footer={draft ?
|
||||
<>
|
||||
<Button onClick={closeProfileDialog} disabled={saving}>i18n:govoplan-files.cancel.77dfd213</Button>
|
||||
<span className="disabled-action-tooltip" data-tooltip={profileSaveTooltip}>
|
||||
<DisabledActionTooltip reason={profileSaveTooltip}>
|
||||
<Button variant="primary" onClick={() => void saveDraft()} disabled={profileSaveDisabled}>
|
||||
<Save size={16} aria-hidden="true" /> {saving ? "i18n:govoplan-files.saving.ae7e8875" : "i18n:govoplan-files.save_connection.07796d38"}
|
||||
</Button>
|
||||
</span>
|
||||
</DisabledActionTooltip>
|
||||
</> :
|
||||
null}>
|
||||
|
||||
@@ -1087,7 +1098,7 @@ export default function FileConnectorSettingsPanel({
|
||||
<h3>Connection</h3>
|
||||
<p>Name the file server connection and choose the provider.</p>
|
||||
</header>
|
||||
<div className="form-grid two-column-form-grid">
|
||||
<div className="form-grid two">
|
||||
<FormField label="i18n:govoplan-files.profile_id.25961d68">
|
||||
<input className={!editingProfileId && !draft.id.trim() ? "field-input-missing" : undefined} aria-invalid={!editingProfileId && !draft.id.trim() || undefined} value={draft.id} disabled={Boolean(editingProfileId) || saving} onChange={(event) => patchDraft({ id: event.target.value })} placeholder={`${scopeType}-webdav`} />
|
||||
</FormField>
|
||||
@@ -1120,7 +1131,7 @@ export default function FileConnectorSettingsPanel({
|
||||
<h3>Location and credentials</h3>
|
||||
<p>Only fields relevant for the selected provider and credential mode are shown.</p>
|
||||
</header>
|
||||
<div className="form-grid two-column-form-grid">
|
||||
<div className="form-grid two">
|
||||
<FormField label="i18n:govoplan-files.endpoint_url.65aaaa45">
|
||||
<input value={draft.endpointUrl} disabled={saving} onChange={(event) => patchDraft({ endpointUrl: event.target.value })} placeholder={ENDPOINT_PLACEHOLDERS[draft.provider]} />
|
||||
</FormField>
|
||||
@@ -1157,7 +1168,7 @@ export default function FileConnectorSettingsPanel({
|
||||
</DismissibleAlert>
|
||||
}
|
||||
<AdvancedOptionsPanel title="Protocol and compatibility options" summary="Change these only when the provider or network requires an override.">
|
||||
<div className="form-grid two-column-form-grid">
|
||||
<div className="form-grid two">
|
||||
<FormField label="i18n:govoplan-files.browse_protocol.d1f27c90">
|
||||
<select value={draft.browseProtocol} disabled={saving} onChange={(event) => patchDraft({ browseProtocol: event.target.value })}>
|
||||
<option value="">i18n:govoplan-files.native_default.ba32f7b6</option>
|
||||
@@ -1191,7 +1202,7 @@ export default function FileConnectorSettingsPanel({
|
||||
<ToggleSwitch label="i18n:govoplan-files.import.d6fbc9d2" checked={draft.canImport} disabled={saving} onChange={(canImport) => patchDraft({ canImport })} />
|
||||
<ToggleSwitch label="i18n:govoplan-files.sync.905f6309" checked={draft.canSync} disabled={saving} onChange={(canSync) => patchDraft({ canSync })} />
|
||||
</div>
|
||||
<div className="form-grid two-column-form-grid">
|
||||
<div className="form-grid two">
|
||||
<FormField label="i18n:govoplan-files.policy.bb9cf141">
|
||||
<select value={draft.policyMode} disabled={saving} onChange={(event) => patchDraft({ policyMode: event.target.value as PolicyMode })}>
|
||||
<option value="allow-provider">i18n:govoplan-files.can_use_this_connection.5091a74c</option>
|
||||
@@ -1342,8 +1353,6 @@ function emptyDraft(scopeType: ConnectorScope): ConnectorProfileDraft {
|
||||
username: "",
|
||||
password: "",
|
||||
token: "",
|
||||
passwordEnv: "",
|
||||
tokenEnv: "",
|
||||
secretRef: "",
|
||||
canBrowse: true,
|
||||
canImport: true,
|
||||
@@ -1406,8 +1415,6 @@ function emptyCredentialDraft(scopeType: ConnectorScope): ConnectorCredentialDra
|
||||
username: "",
|
||||
password: "",
|
||||
token: "",
|
||||
passwordEnv: "",
|
||||
tokenEnv: "",
|
||||
secretRef: "",
|
||||
policyMode: "allow-provider",
|
||||
allowedPaths: "",
|
||||
|
||||
@@ -8,6 +8,7 @@ import {
|
||||
FileDropZone,
|
||||
FormField,
|
||||
LoadingIndicator,
|
||||
ResourceAccessExplanation,
|
||||
ToggleSwitch,
|
||||
hasScope,
|
||||
type ApiSettings,
|
||||
@@ -182,6 +183,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
}, [visibleFiles, folders, currentFolder, searchActive, sortColumn, sortDirection]);
|
||||
const fileListViewportRef = useRef<HTMLDivElement | null>(null);
|
||||
const fileListMeasureRowRef = useRef<HTMLDivElement | null>(null);
|
||||
const managedSpaceLoadsInFlightRef = useRef<Set<string>>(new Set());
|
||||
const [fileListViewportHeight, setFileListViewportHeight] = useState(0);
|
||||
const [fileListScrollTop, setFileListScrollTop] = useState(0);
|
||||
const [fileListRowHeight, setFileListRowHeight] = useState(DEFAULT_FILE_LIST_ROW_HEIGHT);
|
||||
@@ -291,8 +293,9 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
try {
|
||||
const response = await listFileSpaces(settings);
|
||||
setSpaces(response.spaces);
|
||||
setActiveSpaceId((current) => current || response.spaces[0]?.id || "");
|
||||
await Promise.all(response.spaces.filter((space) => !isConnectorSpace(space)).map((space) => loadSpaceContents(space, { silent: true })));
|
||||
const nextActiveSpace = response.spaces.find((space) => space.id === activeSpaceId) ?? response.spaces[0] ?? null;
|
||||
setActiveSpaceId(nextActiveSpace?.id || "");
|
||||
if (nextActiveSpace && !isConnectorSpace(nextActiveSpace)) await loadSpaceContents(nextActiveSpace, { silent: true });
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : String(err));
|
||||
} finally {
|
||||
@@ -331,6 +334,8 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
await loadConnectorSpaceContents(space, { silent: options.silent, folderPath: "" });
|
||||
return;
|
||||
}
|
||||
if (managedSpaceLoadsInFlightRef.current.has(space.id)) return;
|
||||
managedSpaceLoadsInFlightRef.current.add(space.id);
|
||||
if (!options.silent) {
|
||||
setBusy(true);
|
||||
setError("");
|
||||
@@ -372,6 +377,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : String(err));
|
||||
} finally {
|
||||
managedSpaceLoadsInFlightRef.current.delete(space.id);
|
||||
if (!options.silent) setBusy(false);
|
||||
}
|
||||
}
|
||||
@@ -411,6 +417,8 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
const nextSpace = spaces.find((space) => space.id === activeSpaceId);
|
||||
if (nextSpace && isConnectorSpace(nextSpace)) {
|
||||
void loadConnectorSpaceContents(nextSpace, { folderPath: "", silent: true });
|
||||
} else if (nextSpace && filesBySpace[nextSpace.id] === undefined && foldersBySpace[nextSpace.id] === undefined) {
|
||||
void loadSpaceContents(nextSpace, { silent: true });
|
||||
}
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [activeSpaceId, spaces]);
|
||||
@@ -432,6 +440,16 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
return activeDialogTarget;
|
||||
}
|
||||
|
||||
function uploadRejectedReason(target: FileActionTarget | null): string {
|
||||
if (busy || uploadActive) return "Another file operation is already running. Wait until it finishes before dropping files.";
|
||||
if (!canUpload) return "You do not have permission to upload files here.";
|
||||
if (!target) return "Choose a destination folder before dropping files.";
|
||||
const targetSpace = findSpace(target.spaceId);
|
||||
if (!targetSpace) return "The selected upload destination is no longer available.";
|
||||
if (isConnectorSpace(targetSpace)) return "Files cannot be uploaded into connector spaces from this view.";
|
||||
return "The dropped item cannot be uploaded here.";
|
||||
}
|
||||
|
||||
function openDialog(kind: DialogKind, target: FileActionTarget | null = null) {
|
||||
if ((kind === "upload" || kind === "connector-sync") && !canUpload) return;
|
||||
if (kind === "connector-space" && !canOrganize) return;
|
||||
@@ -608,10 +626,12 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
|
||||
|
||||
async function handleFilesUpload(fileList: FileList | File[], options: {conflictStrategy?: ConflictStrategy;conflictResolutions?: ConflictResolution[];bypassConflictDialog?: boolean;target?: FileActionTarget;} = {}) {
|
||||
if (!canUpload) return;
|
||||
const target = options.target ?? currentActionTarget();
|
||||
const targetSpace = target ? findSpace(target.spaceId) : null;
|
||||
if (!target || !targetSpace || isConnectorSpace(targetSpace)) return;
|
||||
if (busy || uploadActive || !canUpload || !target || !targetSpace || isConnectorSpace(targetSpace)) {
|
||||
setError(uploadRejectedReason(target));
|
||||
return;
|
||||
}
|
||||
const selected = Array.from(fileList);
|
||||
if (selected.length === 0) return;
|
||||
if (!options.bypassConflictDialog && !unpackZip) {
|
||||
@@ -671,13 +691,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
}
|
||||
|
||||
async function uploadExternalFilesToTarget(fileList: FileList | File[], target: FileActionTarget) {
|
||||
const previousTarget = dialogTarget;
|
||||
setDialogTarget(target);
|
||||
try {
|
||||
await handleFilesUpload(fileList);
|
||||
} finally {
|
||||
setDialogTarget(previousTarget);
|
||||
}
|
||||
await handleFilesUpload(fileList, { target });
|
||||
}
|
||||
|
||||
async function openConnectorSyncDialog(target: FileActionTarget | null = toolbarTarget()) {
|
||||
@@ -1332,8 +1346,20 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
setDropTargetKey(reason || noop ? "" : dropTargetId(target));
|
||||
return;
|
||||
}
|
||||
event.dataTransfer.dropEffect = canUpload ? "copy" : "none";
|
||||
setDropTargetKey(canUpload ? dropTargetId(target) : "");
|
||||
event.dataTransfer.dropEffect = canUpload && !busy && !uploadActive ? "copy" : "none";
|
||||
setDropTargetKey(canUpload && !busy && !uploadActive ? dropTargetId(target) : "");
|
||||
}
|
||||
|
||||
function handleCurrentFolderDragOver(event: ReactDragEvent<HTMLElement>) {
|
||||
const target = toolbarTarget();
|
||||
if (!target) {
|
||||
event.preventDefault();
|
||||
event.stopPropagation();
|
||||
event.dataTransfer.dropEffect = "none";
|
||||
setDropTargetKey("");
|
||||
return;
|
||||
}
|
||||
handleDropTargetDragOver(event, target);
|
||||
}
|
||||
|
||||
async function handleDropOnTarget(event: ReactDragEvent<HTMLElement>, target: FileActionTarget) {
|
||||
@@ -1341,7 +1367,11 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
event.stopPropagation();
|
||||
setDragActive(false);
|
||||
setDropTargetKey("");
|
||||
if (isConnectorSpace(findSpace(target.spaceId))) return;
|
||||
const hasDroppedFiles = event.dataTransfer.files.length > 0;
|
||||
if (isConnectorSpace(findSpace(target.spaceId))) {
|
||||
if (hasDroppedFiles) setError(uploadRejectedReason(target));
|
||||
return;
|
||||
}
|
||||
if (isInternalDrag(event)) {
|
||||
if (!canOrganize) return;
|
||||
const state = internalDrag ?? parseDragState(event.dataTransfer.getData(INTERNAL_DRAG_TYPE));
|
||||
@@ -1354,9 +1384,27 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
await runTransfer(event.ctrlKey ? "copy" : "move", state.sourceSpaceId, { fileIds: new Set(state.fileIds), folderPaths: new Set(state.folderPaths) }, target);
|
||||
return;
|
||||
}
|
||||
if (canUpload && event.dataTransfer.files.length > 0) {
|
||||
await uploadExternalFilesToTarget(event.dataTransfer.files, target);
|
||||
if (hasDroppedFiles) {
|
||||
if (!canUpload || busy || uploadActive) {
|
||||
setError(uploadRejectedReason(target));
|
||||
return;
|
||||
}
|
||||
await uploadExternalFilesToTarget(event.dataTransfer.files, target);
|
||||
return;
|
||||
}
|
||||
setError("Drop one or more files from your computer, or drag files and folders from this file space.");
|
||||
}
|
||||
|
||||
async function handleDropOnCurrentFolder(event: ReactDragEvent<HTMLElement>) {
|
||||
const target = toolbarTarget();
|
||||
if (!target) {
|
||||
event.preventDefault();
|
||||
event.stopPropagation();
|
||||
setDropTargetKey("");
|
||||
setError(uploadRejectedReason(null));
|
||||
return;
|
||||
}
|
||||
await handleDropOnTarget(event, target);
|
||||
}
|
||||
|
||||
function clearDropState() {
|
||||
@@ -1933,8 +1981,11 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
|
||||
}
|
||||
|
||||
const currentFolderDropTarget = toolbarTarget();
|
||||
const currentFolderDropActive = currentFolderDropTarget ? dropTargetKey === dropTargetId(currentFolderDropTarget) : false;
|
||||
|
||||
return (
|
||||
<div className="workspace-data-page module-entry-page file-manager-page file-manager-fullscreen">
|
||||
<div className="workspace-data-page module-entry-page file-manager-page file-manager-fullscreen files-page">
|
||||
{error &&
|
||||
<DismissibleAlert tone="danger" resetKey={error} floating>{error}</DismissibleAlert>
|
||||
}
|
||||
@@ -2044,9 +2095,12 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
renderConnectorSpaceContent() :
|
||||
|
||||
<div
|
||||
className="file-list-drop-target"
|
||||
className={`file-list-drop-target ${currentFolderDropActive ? "is-drop-target" : ""}`}
|
||||
ref={fileListViewportRef}
|
||||
onScroll={(event) => setFileListScrollTop(event.currentTarget.scrollTop)}
|
||||
onDragOver={handleCurrentFolderDragOver}
|
||||
onDragLeave={clearDropState}
|
||||
onDrop={(event) => void handleDropOnCurrentFolder(event)}
|
||||
onContextMenu={(event) => openContextMenu(event, "empty")}
|
||||
onClick={(event) => {
|
||||
if (event.target === event.currentTarget) applySelectionKeys(new Set<EntrySelectionKey>());
|
||||
@@ -2158,8 +2212,8 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
}
|
||||
|
||||
{accessExplanationTarget &&
|
||||
<FileDialog title="i18n:govoplan-files.access_explanation.75ee7f62" onClose={() => {if (!resourceAccessLoading) {setAccessExplanationTarget(null);setResourceAccessExplanation(null);}}}>
|
||||
<ResourceAccessExplanationContent
|
||||
<FileDialog title="i18n:govoplan-core.access_explanation.75ee7f62" onClose={() => {if (!resourceAccessLoading) {setAccessExplanationTarget(null);setResourceAccessExplanation(null);}}}>
|
||||
<ResourceAccessExplanation
|
||||
loading={resourceAccessLoading}
|
||||
explanation={resourceAccessExplanation}
|
||||
fallbackResourceLabel={accessExplanationTarget.label} />
|
||||
@@ -2172,13 +2226,14 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
{dialog === "upload" &&
|
||||
<FileDialog title={i18nMessage("i18n:govoplan-files.upload_to_value_value.b83a34b4", { value0: activeDialogSpace?.label || "i18n:govoplan-files.files.6ce6c512", value1: activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5" })} onClose={closeDialog}>
|
||||
<FileDropZone
|
||||
disabled={busy || !activeDialogSpace || !canUpload}
|
||||
disabled={busy || !activeDialogTarget || !activeDialogSpace || isConnectorSpace(activeDialogSpace) || !canUpload}
|
||||
busy={uploadActive}
|
||||
progress={visibleUploadProgress}
|
||||
busyLabel={uploadBusyLabel}
|
||||
progressLabel={uploadProgressLabel}
|
||||
note={`Files are uploaded into ${activeDialogTarget?.folderPath || "i18n:govoplan-files.root.e96857c5"}.`}
|
||||
onFiles={(files) => handleFilesUpload(files)} />
|
||||
onRejectedDrop={(reason) => setError(reason === "disabled" ? uploadRejectedReason(activeDialogTarget) : "The browser did not provide readable file data for this drop. Use the file picker, or drag local files from a file manager that exposes file contents to the browser.")}
|
||||
onFiles={(files) => handleFilesUpload(files, { target: activeDialogTarget || undefined })} />
|
||||
|
||||
<ToggleSwitch label="i18n:govoplan-files.unpack_zip_uploads.256fead4" checked={unpackZip} onChange={setUnpackZip} disabled={busy} />
|
||||
<div className="field-block">
|
||||
@@ -2379,7 +2434,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
{dialog === "transfer" && transferDialogState &&
|
||||
<FileDialog title={`${transferMode === "copy" ? "i18n:govoplan-files.copy.af74f7c5" : "i18n:govoplan-files.move.76cdb950"} selection`} onClose={closeDialog}>
|
||||
<p className="muted">i18n:govoplan-files.choose_a_destination_space_and_folder_folders_ar.45158643</p>
|
||||
<div className="form-grid two-column-form-grid">
|
||||
<div className="form-grid two">
|
||||
<FormField label="i18n:govoplan-files.destination_space.92b63970" help="i18n:govoplan-files.select_the_space_that_will_receive_the_selected_.0a8bea8f">
|
||||
<select value={transferDialogState.targetSpaceId} onChange={(event) => setTransferDialogState((current) => current ? { ...current, targetSpaceId: event.target.value, targetFolder: "" } : current)}>
|
||||
{spaces.filter((space) => !isConnectorSpace(space)).map((space) => <option key={space.id} value={space.id}>{space.label}</option>)}
|
||||
@@ -2418,7 +2473,7 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
{dialog === "rename" &&
|
||||
<FileDialog title="i18n:govoplan-files.bulk_rename_selected_items.5e79d694" onClose={closeDialog}>
|
||||
<p className="muted">i18n:govoplan-files.bulk_rename_changes_managed_display_paths_only_i.8cf34a6b</p>
|
||||
<div className="form-grid two-column-form-grid">
|
||||
<div className="form-grid two">
|
||||
<FormField label="i18n:govoplan-files.mode.a7b93d21" help="i18n:govoplan-files.choose_how_the_selected_names_should_be_changed.0231854f">
|
||||
<select value={renameMode} onChange={(event) => setRenameMode(event.target.value as RenameMode)}>
|
||||
<option value="prefix">i18n:govoplan-files.add_prefix.672452bc</option>
|
||||
@@ -2454,71 +2509,6 @@ export default function FilesPage({ settings, auth }: {settings: ApiSettings;aut
|
||||
|
||||
}
|
||||
|
||||
function ResourceAccessExplanationContent({
|
||||
loading,
|
||||
explanation,
|
||||
fallbackResourceLabel
|
||||
}: {loading: boolean;explanation: ResourceAccessExplanationResponse | null;fallbackResourceLabel: string;}) {
|
||||
if (loading) return <p className="muted small-note">i18n:govoplan-files.loading_access_explanation.04a7c934</p>;
|
||||
if (!explanation) return null;
|
||||
const userLabel = explanation.user.display_name || explanation.user.email || explanation.user.id;
|
||||
const resourceLabel = explanation.provenance.find((item) => item.kind === "resource")?.label || fallbackResourceLabel || explanation.resource_id;
|
||||
return (
|
||||
<>
|
||||
<div className="form-grid compact responsive-form-grid">
|
||||
<div><span className="form-label">i18n:govoplan-files.user.9f8a2389</span><p>{userLabel}</p></div>
|
||||
<div><span className="form-label">i18n:govoplan-files.resource.d1c626a9</span><p>{resourceLabel}</p></div>
|
||||
<div><span className="form-label">i18n:govoplan-files.action.97c89a4d</span><p><code>{explanation.action}</code></p></div>
|
||||
<div><span className="form-label">i18n:govoplan-files.evidence.8487d192</span><p>{explanation.provenance.length}</p></div>
|
||||
</div>
|
||||
{explanation.provenance.length === 0 ?
|
||||
<p className="muted small-note">i18n:govoplan-files.no_access_evidence_was_returned.84a21e4e</p> :
|
||||
<div className="admin-assignment-grid">
|
||||
{explanation.provenance.map((item, index) =>
|
||||
<div key={`${item.kind}:${item.id ?? index}`}>
|
||||
<strong>{provenanceKindLabel(item.kind)}</strong>
|
||||
<div className="muted small-note">
|
||||
{item.source || "i18n:govoplan-files.no_source.6dcf9723"}
|
||||
{item.id && <> · <code>{item.id}</code></>}
|
||||
</div>
|
||||
{item.label && <p>{item.label}</p>}
|
||||
{Object.keys(item.details ?? {}).length > 0 && <p className="muted small-note">{formatProvenanceDetails(item.details ?? {})}</p>}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
}
|
||||
</>);
|
||||
}
|
||||
|
||||
function provenanceKindLabel(kind: string): string {
|
||||
switch (kind) {
|
||||
case "resource":
|
||||
return "i18n:govoplan-files.resource.d1c626a9";
|
||||
case "owner":
|
||||
return "i18n:govoplan-files.owner.89ff3122";
|
||||
case "share":
|
||||
return "i18n:govoplan-files.share.09ca55ca";
|
||||
case "policy":
|
||||
return "i18n:govoplan-files.policy.0b779a05";
|
||||
case "role":
|
||||
return "i18n:govoplan-files.role.b5b4a5a2";
|
||||
case "right":
|
||||
return "i18n:govoplan-files.permission.2f81a22d";
|
||||
default:
|
||||
return kind;
|
||||
}
|
||||
}
|
||||
|
||||
function formatProvenanceDetails(details: Record<string, unknown>): string {
|
||||
return Object.entries(details).map(([key, value]) => `${key}: ${formatProvenanceValue(value)}`).join("; ");
|
||||
}
|
||||
|
||||
function formatProvenanceValue(value: unknown): string {
|
||||
if (value === null || value === undefined) return "i18n:govoplan-files.none.6eef6648";
|
||||
if (typeof value === "string" || typeof value === "number" || typeof value === "boolean") return String(value);
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
|
||||
function FileSearchRow({
|
||||
value,
|
||||
caseSensitive,
|
||||
|
||||
@@ -252,7 +252,6 @@ export const generatedTranslations: PlatformTranslations = {
|
||||
"i18n:govoplan-files.overwrite.0625c54e": "Overwrite",
|
||||
"i18n:govoplan-files.owner_space.364c4b62": "Owner space",
|
||||
"i18n:govoplan-files.parent_folder.d8ed4c2a": "Parent folder",
|
||||
"i18n:govoplan-files.password_environment_variable.0e77673f": "Password environment variable",
|
||||
"i18n:govoplan-files.password.8be3c943": "Password",
|
||||
"i18n:govoplan-files.path_limited.d32cbef0": "Path-limited",
|
||||
"i18n:govoplan-files.pattern_preview_results.56cea56c": "Pattern preview results",
|
||||
@@ -318,7 +317,6 @@ export const generatedTranslations: PlatformTranslations = {
|
||||
"i18n:govoplan-files.this_file_credential.695efb90": "this file credential",
|
||||
"i18n:govoplan-files.this_folder_is_empty_upload_files_in_the_top_lev.cb6c3a1a": "This folder is empty. Upload files in the top-level Files module.",
|
||||
"i18n:govoplan-files.this_folder_is_empty_use_upload_or_create_folder.5977d784": "This folder is empty. Use Upload or Create Folder to add content.",
|
||||
"i18n:govoplan-files.token_environment_variable.5af4a79e": "Token environment variable",
|
||||
"i18n:govoplan-files.token.a1141eb9": "Token",
|
||||
"i18n:govoplan-files.unpack_zip_uploads.256fead4": "Unpack ZIP uploads",
|
||||
"i18n:govoplan-files.unpacking_zip_archive.698095f4": "Unpacking ZIP archive",
|
||||
@@ -609,7 +607,6 @@ export const generatedTranslations: PlatformTranslations = {
|
||||
"i18n:govoplan-files.overwrite.0625c54e": "Overwrite",
|
||||
"i18n:govoplan-files.owner_space.364c4b62": "Owner space",
|
||||
"i18n:govoplan-files.parent_folder.d8ed4c2a": "Parent folder",
|
||||
"i18n:govoplan-files.password_environment_variable.0e77673f": "Password environment variable",
|
||||
"i18n:govoplan-files.password.8be3c943": "Passwort",
|
||||
"i18n:govoplan-files.path_limited.d32cbef0": "Path-limited",
|
||||
"i18n:govoplan-files.pattern_preview_results.56cea56c": "Pattern preview results",
|
||||
@@ -675,7 +672,6 @@ export const generatedTranslations: PlatformTranslations = {
|
||||
"i18n:govoplan-files.this_file_credential.695efb90": "this file credential",
|
||||
"i18n:govoplan-files.this_folder_is_empty_upload_files_in_the_top_lev.cb6c3a1a": "This folder is empty. Upload files in the top-level Files module.",
|
||||
"i18n:govoplan-files.this_folder_is_empty_use_upload_or_create_folder.5977d784": "This folder is empty. Use Upload or Create Folder to add content.",
|
||||
"i18n:govoplan-files.token_environment_variable.5af4a79e": "Token environment variable",
|
||||
"i18n:govoplan-files.token.a1141eb9": "Token",
|
||||
"i18n:govoplan-files.unpack_zip_uploads.256fead4": "Unpack ZIP uploads",
|
||||
"i18n:govoplan-files.unpacking_zip_archive.698095f4": "Unpacking ZIP archive",
|
||||
|
||||
@@ -1,85 +1,6 @@
|
||||
/* Files manager */
|
||||
.file-manager-page.file-manager-fullscreen {
|
||||
position: relative;
|
||||
display: grid;
|
||||
grid-template-rows: 1fr;
|
||||
height: calc(100vh - 115px);
|
||||
padding: 0;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.file-manager-toolbar {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
flex-wrap: wrap;
|
||||
padding: 10px 14px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
background: var(--panel-header);
|
||||
}
|
||||
|
||||
.file-manager-toolbar .button {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 7px;
|
||||
}
|
||||
|
||||
.file-manager-shell {
|
||||
display: grid;
|
||||
.files-page .file-manager-shell {
|
||||
grid-template-columns: minmax(240px, 300px) minmax(0, 1fr);
|
||||
min-height: 0;
|
||||
height: 100%;
|
||||
border: 1px solid var(--line);
|
||||
border-radius: 0;
|
||||
overflow: hidden;
|
||||
background: var(--panel);
|
||||
}
|
||||
|
||||
.file-list-panel {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
min-width: 0;
|
||||
min-height: 0;
|
||||
background: var(--panel);
|
||||
}
|
||||
|
||||
.file-list-sticky {
|
||||
flex: 0 0 auto;
|
||||
z-index: 2;
|
||||
border-bottom: 1px solid var(--line);
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
|
||||
.file-breadcrumbs {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
flex-wrap: wrap;
|
||||
gap: 6px;
|
||||
min-height: 32px;
|
||||
padding: 10px 14px 6px;
|
||||
}
|
||||
|
||||
.file-breadcrumb,
|
||||
.file-breadcrumb-segment {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 5px;
|
||||
}
|
||||
|
||||
.file-breadcrumb {
|
||||
border: 0;
|
||||
background: transparent;
|
||||
color: var(--text-strong);
|
||||
cursor: pointer;
|
||||
font-weight: 800;
|
||||
padding: 4px 6px;
|
||||
border-radius: 7px;
|
||||
}
|
||||
|
||||
.file-breadcrumb:hover:not(:disabled),
|
||||
.file-breadcrumb:focus-visible {
|
||||
background: var(--panel);
|
||||
outline: none;
|
||||
}
|
||||
|
||||
.file-search-row {
|
||||
@@ -90,53 +11,21 @@
|
||||
padding: 4px 0 10px 14px;
|
||||
}
|
||||
|
||||
.file-list-meta {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 12px;
|
||||
flex-wrap: wrap;
|
||||
padding: 8px 14px;
|
||||
border-top: 1px solid var(--line);
|
||||
color: var(--muted);
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.file-list-drop-target {
|
||||
position: relative;
|
||||
flex: 1 1 auto;
|
||||
min-height: 0;
|
||||
overflow: auto;
|
||||
transition: background-color .16s ease, box-shadow .16s ease;
|
||||
}
|
||||
|
||||
.file-list-drop-target.is-active,
|
||||
.file-list-drop-target.is-drop-target {
|
||||
background: var(--line);
|
||||
box-shadow: inset 0 0 0 2px var(--line-dark);
|
||||
}
|
||||
|
||||
.file-list-table {
|
||||
.files-page .file-list-table {
|
||||
min-width: 760px;
|
||||
}
|
||||
|
||||
.file-list-table-head,
|
||||
.file-list-row {
|
||||
display: grid;
|
||||
.files-page .file-list-table-head,
|
||||
.files-page .file-list-row,
|
||||
.managed-pattern-results .file-list-table-head,
|
||||
.managed-pattern-results .file-list-row {
|
||||
grid-template-columns: minmax(280px, 1fr) 120px 240px;
|
||||
gap: 12px;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.file-list-table-head {
|
||||
padding: 10px 14px;
|
||||
border-top: 1px solid var(--line);
|
||||
background: var(--panel);
|
||||
color: var(--muted);
|
||||
font-size: 12px;
|
||||
font-weight: 800;
|
||||
letter-spacing: .04em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.file-list-table-head button {
|
||||
@@ -158,24 +47,6 @@
|
||||
outline: none;
|
||||
}
|
||||
|
||||
.file-list-row {
|
||||
min-height: 58px;
|
||||
padding: 9px 14px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
cursor: default;
|
||||
user-select: none;
|
||||
}
|
||||
|
||||
.file-list-row:focus-visible {
|
||||
outline: 2px solid var(--line-dark);
|
||||
outline-offset: -2px;
|
||||
}
|
||||
|
||||
.file-list-row:hover,
|
||||
.file-list-row.is-selected {
|
||||
background: var(--line);
|
||||
}
|
||||
|
||||
.file-list-row.is-drop-target {
|
||||
background: var(--line);
|
||||
box-shadow: inset 0 0 0 2px var(--line-dark);
|
||||
@@ -191,43 +62,6 @@
|
||||
background: transparent;
|
||||
}
|
||||
|
||||
.file-list-name-cell {
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.file-list-name {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
max-width: 100%;
|
||||
min-width: 0;
|
||||
border: 0;
|
||||
background: transparent;
|
||||
color: var(--text);
|
||||
cursor: default;
|
||||
font: inherit;
|
||||
text-align: left;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.file-list-name > span {
|
||||
display: grid;
|
||||
min-width: 0;
|
||||
gap: 2px;
|
||||
}
|
||||
|
||||
.file-list-name strong,
|
||||
.file-list-name small {
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.file-list-name small {
|
||||
color: var(--muted);
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.file-linkage-status {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
@@ -238,29 +72,8 @@
|
||||
color: var(--text-strong);
|
||||
}
|
||||
|
||||
.file-row-icon {
|
||||
color: var(--muted);
|
||||
flex: 0 0 auto;
|
||||
}
|
||||
|
||||
.folder-row .file-row-icon {
|
||||
color: #b6791d;
|
||||
}
|
||||
|
||||
.file-row-tail {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 10px;
|
||||
min-width: 0;
|
||||
color: var(--muted);
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.file-list-empty {
|
||||
padding: 36px 20px;
|
||||
color: var(--muted);
|
||||
text-align: center;
|
||||
color: var(--warning-deep);
|
||||
}
|
||||
|
||||
.file-list-virtual-spacer {
|
||||
@@ -274,7 +87,7 @@
|
||||
display: grid;
|
||||
min-width: 190px;
|
||||
overflow: hidden;
|
||||
border: 1px solid var(--line-dark);
|
||||
border: var(--border-line-dark);
|
||||
border-radius: 12px;
|
||||
background: var(--panel);
|
||||
box-shadow: var(--shadow-popover);
|
||||
@@ -299,7 +112,7 @@
|
||||
|
||||
.file-context-menu button:hover,
|
||||
.file-context-menu button:focus-visible {
|
||||
background: rgba(13, 110, 253, .08);
|
||||
background: var(--primary-soft);
|
||||
outline: none;
|
||||
}
|
||||
|
||||
@@ -315,14 +128,14 @@
|
||||
display: grid;
|
||||
place-items: center;
|
||||
padding: 22px;
|
||||
background: rgba(28, 25, 22, .38);
|
||||
background: var(--dialog-backdrop);
|
||||
}
|
||||
|
||||
.file-dialog {
|
||||
width: min(620px, 100%);
|
||||
max-height: min(720px, calc(100vh - 44px));
|
||||
overflow: auto;
|
||||
border: 1px solid var(--line-dark);
|
||||
border: var(--border-line-dark);
|
||||
border-radius: 16px;
|
||||
background: var(--panel);
|
||||
box-shadow: var(--shadow-popover);
|
||||
@@ -334,7 +147,7 @@
|
||||
justify-content: space-between;
|
||||
gap: 12px;
|
||||
padding: 15px 18px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
border-bottom: var(--border-line);
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
|
||||
@@ -397,7 +210,7 @@
|
||||
overflow-y: auto;
|
||||
scrollbar-gutter: stable;
|
||||
padding: 8px;
|
||||
border: 1px solid var(--line);
|
||||
border: var(--border-line);
|
||||
border-radius: 12px;
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
@@ -428,13 +241,13 @@
|
||||
.file-folder-selector-node:hover:not(:disabled),
|
||||
.file-folder-selector-node:focus-visible,
|
||||
.file-folder-selector-node.is-selected {
|
||||
background: rgba(13, 110, 253, .09);
|
||||
background: var(--primary-soft);
|
||||
outline: none;
|
||||
}
|
||||
|
||||
.file-folder-selector-node.is-selected {
|
||||
color: var(--text-strong);
|
||||
box-shadow: inset 0 0 0 1px rgba(13, 110, 253, .25);
|
||||
box-shadow: var(--primary-inset-ring);
|
||||
}
|
||||
|
||||
.file-folder-selector-empty {
|
||||
@@ -462,7 +275,7 @@
|
||||
grid-template-rows: auto auto minmax(0, 1fr);
|
||||
min-height: 320px;
|
||||
overflow: hidden;
|
||||
border: 1px solid var(--line);
|
||||
border: var(--border-line);
|
||||
border-radius: 12px;
|
||||
background: var(--panel);
|
||||
}
|
||||
@@ -473,7 +286,7 @@
|
||||
gap: 10px;
|
||||
align-items: center;
|
||||
padding: 10px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
border-bottom: var(--border-line);
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
|
||||
@@ -577,7 +390,7 @@
|
||||
gap: 3px;
|
||||
min-width: 0;
|
||||
padding: 10px 12px;
|
||||
border: 1px solid var(--line);
|
||||
border: var(--border-line);
|
||||
border-radius: 10px;
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
@@ -625,7 +438,7 @@
|
||||
gap: 12px;
|
||||
align-items: center;
|
||||
padding: 10px 12px;
|
||||
border: 1px solid var(--line);
|
||||
border: var(--border-line);
|
||||
border-radius: 8px;
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
@@ -698,6 +511,11 @@
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.file-connector-secret-fields {
|
||||
display: grid;
|
||||
gap: 18px;
|
||||
}
|
||||
|
||||
@media (max-width: 760px) {
|
||||
.file-connector-profile-row {
|
||||
grid-template-columns: 1fr;
|
||||
@@ -719,25 +537,27 @@
|
||||
}
|
||||
|
||||
@media (max-width: 1050px) {
|
||||
.file-manager-shell {
|
||||
.files-page .file-manager-shell {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.file-tree-panel {
|
||||
.files-page .file-tree-panel {
|
||||
max-height: 260px;
|
||||
border-right: 0;
|
||||
border-bottom: 1px solid var(--line);
|
||||
border-bottom: var(--border-line);
|
||||
}
|
||||
|
||||
.file-list-table-head {
|
||||
.files-page .file-list-table-head,
|
||||
.managed-pattern-results .file-list-table-head {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.file-list-table {
|
||||
.files-page .file-list-table {
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.file-list-row {
|
||||
.files-page .file-list-row,
|
||||
.managed-pattern-results .file-list-row {
|
||||
grid-template-columns: 1fr;
|
||||
gap: 8px;
|
||||
}
|
||||
@@ -747,28 +567,11 @@
|
||||
}
|
||||
}
|
||||
|
||||
.file-manager-shell.is-loading .file-tree-panel,
|
||||
.file-manager-shell.is-loading .file-list-panel {
|
||||
filter: blur(1.5px);
|
||||
pointer-events: none;
|
||||
user-select: none;
|
||||
}
|
||||
|
||||
.file-manager-loading-overlay {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
z-index: 35;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
background: rgba(255, 255, 255, .12);
|
||||
backdrop-filter: blur(1px);
|
||||
}
|
||||
|
||||
.file-conflict-summary {
|
||||
display: grid;
|
||||
gap: 3px;
|
||||
padding: 12px 14px;
|
||||
border: 1px solid var(--line);
|
||||
border: var(--border-line);
|
||||
border-radius: 12px;
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
@@ -791,7 +594,7 @@
|
||||
gap: 10px;
|
||||
align-items: center;
|
||||
padding: 10px;
|
||||
border: 1px solid var(--line);
|
||||
border: var(--border-line);
|
||||
border-radius: 12px;
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
@@ -842,8 +645,8 @@
|
||||
|
||||
.rename-preview-more:hover,
|
||||
.rename-preview-more:focus-visible {
|
||||
border-color: rgba(13, 110, 253, .45);
|
||||
background: rgba(13, 110, 253, .08);
|
||||
border-color: var(--primary-border);
|
||||
background: var(--primary-soft);
|
||||
color: var(--text-strong);
|
||||
outline: none;
|
||||
}
|
||||
@@ -860,7 +663,7 @@
|
||||
|
||||
.managed-file-chooser-dialog > .dialog-header {
|
||||
padding: 16px 18px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
border-bottom: var(--border-line);
|
||||
background: var(--panel);
|
||||
}
|
||||
|
||||
@@ -891,9 +694,9 @@
|
||||
.managed-file-chooser-footer {
|
||||
flex: 0 0 auto;
|
||||
padding: 12px 18px;
|
||||
border-top: 1px solid var(--line);
|
||||
border-top: var(--border-line);
|
||||
background: var(--panel);
|
||||
box-shadow: 0 -8px 24px rgba(15, 23, 42, .06);
|
||||
box-shadow: var(--shadow-footer);
|
||||
}
|
||||
|
||||
.managed-file-chooser-layout {
|
||||
@@ -911,7 +714,7 @@
|
||||
.managed-file-chooser-spaces {
|
||||
min-width: 0;
|
||||
padding: 16px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-right: var(--border-line);
|
||||
background: var(--panel-soft);
|
||||
overflow: auto;
|
||||
}
|
||||
@@ -955,7 +758,7 @@
|
||||
border: 1px solid transparent;
|
||||
border-radius: 9px;
|
||||
background: transparent;
|
||||
color: var(--ink);
|
||||
color: var(--text-strong);
|
||||
text-align: left;
|
||||
cursor: pointer;
|
||||
}
|
||||
@@ -1014,7 +817,7 @@
|
||||
gap: 12px;
|
||||
align-items: center;
|
||||
padding: 12px 14px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
border-bottom: var(--border-line);
|
||||
}
|
||||
|
||||
.managed-file-breadcrumb {
|
||||
@@ -1033,7 +836,7 @@
|
||||
border: 0;
|
||||
border-radius: 6px;
|
||||
background: transparent;
|
||||
color: var(--ink);
|
||||
color: var(--text-strong);
|
||||
padding: 5px 6px;
|
||||
cursor: pointer;
|
||||
}
|
||||
@@ -1046,7 +849,7 @@
|
||||
display: grid;
|
||||
gap: 8px;
|
||||
padding: 12px 14px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
border-bottom: var(--border-line);
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
|
||||
@@ -1075,7 +878,7 @@
|
||||
align-items: center;
|
||||
min-height: 38px;
|
||||
padding: 0 20px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
border-bottom: var(--border-line);
|
||||
background: var(--panel-soft);
|
||||
color: var(--muted);
|
||||
font-size: 12px;
|
||||
@@ -1132,7 +935,7 @@
|
||||
}
|
||||
|
||||
.managed-file-entry:disabled {
|
||||
color: var(--ink);
|
||||
color: var(--text-strong);
|
||||
opacity: 1;
|
||||
cursor: default;
|
||||
}
|
||||
@@ -1206,7 +1009,7 @@
|
||||
.managed-file-chooser-note {
|
||||
margin: 0;
|
||||
padding: 10px 14px;
|
||||
border-top: 1px solid var(--line);
|
||||
border-top: var(--border-line);
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
|
||||
@@ -1240,7 +1043,7 @@
|
||||
border-right: 0;
|
||||
border-left: 0;
|
||||
background: transparent;
|
||||
color: var(--ink);
|
||||
color: var(--text-strong);
|
||||
text-align: left;
|
||||
font: inherit;
|
||||
cursor: pointer;
|
||||
@@ -1267,27 +1070,6 @@
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.split-field-action {
|
||||
gap: 0;
|
||||
}
|
||||
|
||||
.split-field-action > input,
|
||||
.split-field-action > select,
|
||||
.split-field-action > textarea {
|
||||
border-top-right-radius: 0 !important;
|
||||
border-bottom-right-radius: 0 !important;
|
||||
}
|
||||
|
||||
.split-field-action > button,
|
||||
.split-field-action > .button,
|
||||
.split-field-action > .btn {
|
||||
align-self: stretch;
|
||||
margin-left: -1px;
|
||||
border-top-left-radius: 0;
|
||||
border-bottom-left-radius: 0;
|
||||
box-shadow: none;
|
||||
}
|
||||
|
||||
@media (max-width: 850px) {
|
||||
.managed-file-entry-head,
|
||||
.managed-file-entry {
|
||||
@@ -1313,7 +1095,7 @@
|
||||
|
||||
.managed-file-chooser-spaces {
|
||||
border-right: 0;
|
||||
border-bottom: 1px solid var(--line);
|
||||
border-bottom: var(--border-line);
|
||||
max-height: 160px;
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user