add-ideas/govoplan-files
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

chore: sync GovOPlaN module split state

Browse Source
This commit is contained in:
Albrecht Degering
2026-07-10 12:51:21 +02:00
parent 18e6c3eb9b
commit 13fd7fc3bd
50 changed files with 12678 additions and 1109 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
dev/connectors/.env.example Normal file
View File

@@ -0,0 +1,19 @@
NEXTCLOUD_HOST_PORT=9081
NEXTCLOUD_DB_ROOT_PASSWORD=govoplan-nextcloud-root
NEXTCLOUD_DB_PASSWORD=govoplan-nextcloud
NEXTCLOUD_ADMIN_USER=admin
NEXTCLOUD_ADMIN_PASSWORD=govoplan-nextcloud-admin
SEAFILE_HOST_PORT=9082
SEAFILE_MYSQL_ROOT_PASSWORD=govoplan-seafile-root
SEAFILE_ADMIN_EMAIL=admin@example.local
SEAFILE_ADMIN_PASSWORD=govoplan-seafile-admin
WEBDAV_HOST_PORT=9083
WEBDAV_USER=govoplan
WEBDAV_PASSWORD=govoplan-webdav
SMB_HOST_PORT=1445
SMB_SHARE_NAME=files
SMB_USER=govoplan
SMB_PASSWORD=govoplan-smb

130
dev/connectors/README.md Normal file
View File

@@ -0,0 +1,130 @@
# Connector Dev Stack
This directory keeps local Docker Compose assets for GovOPlaN file connector
development. The stack is intentionally separate from production deployment and
binds services to localhost high ports.
## Start
```bash
cd /mnt/DATA/git/govoplan-files/dev/connectors
cp .env.example .env
docker compose up -d nextcloud nextcloud-db webdav smb
docker compose up -d seafile-db seafile-memcached seafile
```
If the SMB service was already running before a compose change, recreate it so
the image and share configuration are applied:
```bash
docker compose rm -sf smb
docker compose up -d --build smb
```
Endpoints:
- Nextcloud: `http://127.0.0.1:9081`, WebDAV root
`http://127.0.0.1:9081/remote.php/dav/files/admin/`
- Seafile: `http://127.0.0.1:9082`, WebDAV root
`http://127.0.0.1:9082/seafdav/`
- WebDAV: `http://127.0.0.1:9083`
- SMB: `smb://127.0.0.1:1445/files`
The local fixture data under `data/` is ignored by git.
## GovOPlaN Profile Config
Connector profiles are read from `GOVOPLAN_FILES_CONNECTOR_PROFILES_JSON` or
`GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE`. Credentials should be referenced by
secret refs or environment variables; profile API responses only expose the
credential source and configured state.
Example local profile file:
```json
{
"profiles": [
{
"id": "dev-seafile",
"label": "Dev Seafile",
"provider": "seafile",
"endpoint_url": "http://127.0.0.1:9082",
"scope_type": "system",
"credential_mode": "basic",
"username": "admin@example.local",
"password_env": "SEAFILE_ADMIN_PASSWORD",
"capabilities": ["browse", "import"],
"metadata": { "webdav_endpoint_url": "http://127.0.0.1:9082/seafdav/" },
"policy": { "allow": { "providers": ["seafile"] } }
},
{
"id": "dev-nextcloud",
"label": "Dev Nextcloud",
"provider": "nextcloud",
"endpoint_url": "http://127.0.0.1:9081/remote.php/dav/files/admin/",
"scope_type": "system",
"credential_mode": "basic",
"username": "admin",
"password_env": "NEXTCLOUD_ADMIN_PASSWORD",
"capabilities": ["browse", "import"],
"policy": { "allow": { "providers": ["nextcloud", "webdav"] } }
},
{
"id": "dev-webdav",
"label": "Dev WebDAV",
"provider": "webdav",
"endpoint_url": "http://127.0.0.1:9083",
"scope_type": "system",
"credential_mode": "basic",
"username": "govoplan",
"password_env": "WEBDAV_PASSWORD",
"capabilities": ["browse", "import"],
"policy": { "allow": { "providers": ["webdav"] } }
},
{
"id": "dev-smb",
"label": "Dev SMB",
"provider": "smb",
"endpoint_url": "smb://127.0.0.1:1445/files",
"scope_type": "system",
"credential_mode": "basic",
"username": "govoplan",
"password_env": "SMB_PASSWORD",
"capabilities": ["browse", "import"],
"policy": { "allow": { "providers": ["smb"] } }
}
]
}
```
Start GovOPlaN with:
```bash
export GOVOPLAN_FILES_CONNECTOR_PROFILES_FILE=/mnt/DATA/git/govoplan-files/dev/connectors/profiles.local.json
```
## Smoke Test
Run the connector helper smoke checks from an environment where
`govoplan-files` and `httpx` are importable:
```bash
cd /mnt/DATA/git/govoplan-files/dev/connectors
/mnt/DATA/git/govoplan-core/.venv/bin/python smoke.py
```
The script reads `.env` from this directory when present, seeds tiny WebDAV,
Nextcloud, and SMB fixtures, then browses and imports them through the connector
helper layer. Pass `--require-smb` after recreating the SMB container to fail on
SMB access errors instead of reporting them as an optional skip.
SMB smoke checks need the optional Python dependency in the environment running
the script:
```bash
/mnt/DATA/git/govoplan-core/.venv/bin/python -m pip install -e /mnt/DATA/git/govoplan-files[smb]
```
The SMB service is built from `dev/connectors/smb/` so the development share is
deterministic: one `files` share backed by `data/smb`, with the credentials from
`.env`.

2
dev/connectors/data/.gitignore vendored Normal file
View File

@@ -0,0 +1,2 @@
*
!.gitignore

106
dev/connectors/docker-compose.yml Normal file
View File

@@ -0,0 +1,106 @@
name: govoplan-files-connectors
services:
nextcloud-db:
image: mariadb:10.11
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
restart: unless-stopped
environment:
MYSQL_ROOT_PASSWORD: ${NEXTCLOUD_DB_ROOT_PASSWORD:-govoplan-nextcloud-root}
MYSQL_DATABASE: nextcloud
MYSQL_USER: nextcloud
MYSQL_PASSWORD: ${NEXTCLOUD_DB_PASSWORD:-govoplan-nextcloud}
volumes:
- nextcloud-db:/var/lib/mysql
nextcloud:
image: nextcloud:apache
restart: unless-stopped
depends_on:
- nextcloud-db
ports:
- "127.0.0.1:${NEXTCLOUD_HOST_PORT:-9081}:80"
environment:
MYSQL_HOST: nextcloud-db
MYSQL_DATABASE: nextcloud
MYSQL_USER: nextcloud
MYSQL_PASSWORD: ${NEXTCLOUD_DB_PASSWORD:-govoplan-nextcloud}
NEXTCLOUD_ADMIN_USER: ${NEXTCLOUD_ADMIN_USER:-admin}
NEXTCLOUD_ADMIN_PASSWORD: ${NEXTCLOUD_ADMIN_PASSWORD:-govoplan-nextcloud-admin}
NEXTCLOUD_TRUSTED_DOMAINS: "localhost 127.0.0.1"
volumes:
- nextcloud:/var/www/html
seafile-db:
image: mariadb:10.11
restart: unless-stopped
environment:
MYSQL_ROOT_PASSWORD: ${SEAFILE_MYSQL_ROOT_PASSWORD:-govoplan-seafile-root}
MARIADB_AUTO_UPGRADE: "1"
MYSQL_LOG_CONSOLE: "true"
volumes:
- seafile-db:/var/lib/mysql
seafile-memcached:
image: memcached:1.6
restart: unless-stopped
command: memcached -m 256
seafile:
image: seafileltd/seafile-mc:11.0-latest
restart: unless-stopped
depends_on:
- seafile-db
- seafile-memcached
ports:
- "127.0.0.1:${SEAFILE_HOST_PORT:-9082}:80"
environment:
DB_HOST: seafile-db
DB_ROOT_PASSWD: ${SEAFILE_MYSQL_ROOT_PASSWORD:-govoplan-seafile-root}
TIME_ZONE: Etc/UTC
SEAFILE_ADMIN_EMAIL: ${SEAFILE_ADMIN_EMAIL:-admin@example.local}
SEAFILE_ADMIN_PASSWORD: ${SEAFILE_ADMIN_PASSWORD:-govoplan-seafile-admin}
SEAFILE_SERVER_LETSENCRYPT: "false"
SEAFILE_SERVER_HOSTNAME: "127.0.0.1:${SEAFILE_HOST_PORT:-9082}"
volumes:
- seafile-data:/shared
webdav:
image: rclone/rclone:latest
restart: unless-stopped
command:
- serve
- webdav
- /data
- --addr
- :8080
- --user
- ${WEBDAV_USER:-govoplan}
- --pass
- ${WEBDAV_PASSWORD:-govoplan-webdav}
ports:
- "127.0.0.1:${WEBDAV_HOST_PORT:-9083}:8080"
volumes:
- ./data/webdav:/data
smb:
build:
context: ./smb
image: govoplan-files-samba-dev:latest
restart: unless-stopped
environment:
SMB_SHARE_NAME: ${SMB_SHARE_NAME:-files}
SMB_USER: ${SMB_USER:-govoplan}
SMB_PASSWORD: ${SMB_PASSWORD:-govoplan-smb}
SMB_UID: ${SMB_UID:-1000}
SMB_GID: ${SMB_GID:-1000}
ports:
- "127.0.0.1:${SMB_HOST_PORT:-1445}:445"
volumes:
- ./data/smb:/storage
volumes:
nextcloud-db:
nextcloud:
seafile-db:
seafile-data:

10
dev/connectors/smb/Dockerfile Normal file
View File

@@ -0,0 +1,10 @@
FROM alpine:3.20
RUN apk add --no-cache samba-server samba-common-tools
COPY entrypoint.sh /usr/local/bin/govoplan-samba-entrypoint
RUN chmod +x /usr/local/bin/govoplan-samba-entrypoint
EXPOSE 445
ENTRYPOINT ["/usr/local/bin/govoplan-samba-entrypoint"]

50
dev/connectors/smb/entrypoint.sh Normal file
View File

@@ -0,0 +1,50 @@
#!/bin/sh
set -eu
share_name="${SMB_SHARE_NAME:-files}"
user_name="${SMB_USER:-govoplan}"
password="${SMB_PASSWORD:-govoplan-smb}"
uid="${SMB_UID:-1000}"
gid="${SMB_GID:-1000}"
mkdir -p /storage /var/lib/samba/private /run/samba
if ! getent group "$user_name" >/dev/null 2>&1; then
addgroup -g "$gid" "$user_name"
fi
if ! id "$user_name" >/dev/null 2>&1; then
adduser -D -H -s /sbin/nologin -u "$uid" -G "$user_name" "$user_name"
fi
chown -R "$user_name:$user_name" /storage
cat > /etc/samba/smb.conf <<EOF
[global]
server role = standalone server
workgroup = WORKGROUP
security = user
map to guest = Never
server min protocol = SMB2
load printers = no
printing = bsd
disable spoolss = yes
log level = 1
passdb backend = tdbsam
[$share_name]
path = /storage
browseable = yes
read only = no
guest ok = no
valid users = $user_name
force user = $user_name
force group = $user_name
create mask = 0664
directory mask = 0775
EOF
printf '%s\n%s\n' "$password" "$password" | smbpasswd -s -a "$user_name" >/dev/null
smbpasswd -e "$user_name" >/dev/null
exec smbd --foreground --no-process-group --debug-stdout

215
dev/connectors/smoke.py Normal file
View File

@@ -0,0 +1,215 @@
from __future__ import annotations
import argparse
import os
import socket
from pathlib import Path
import httpx
from govoplan_files.backend.storage.connector_browse import browse_connector_profile
from govoplan_files.backend.storage.connector_imports import read_connector_file
from govoplan_files.backend.storage.connector_profiles import connector_profiles_from_payload
ROOT = Path(__file__).resolve().parent
def main() -> int:
parser = argparse.ArgumentParser(description="Smoke-test GovOPlaN connector helpers against the local dev compose stack.")
parser.add_argument("--require-smb", action="store_true", help="Fail if the SMB connector cannot browse/import.")
parser.add_argument("--debug-smb", action="store_true", help="Print direct smbclient probes before the connector smoke check.")
args = parser.parse_args()
_load_dotenv()
_default_env()
preflight_failures = _preflight_services(require_smb=args.require_smb)
if preflight_failures:
for failure in preflight_failures:
print(f"FAIL {failure}")
print("Start or recreate the connector stack from this directory with: docker compose up -d nextcloud nextcloud-db webdav smb")
return 1
_seed_webdav_fixture()
_seed_smb_fixture()
try:
_seed_nextcloud_fixture()
except httpx.HTTPError as exc:
print(f"FAIL dev-nextcloud seed failed: {exc}")
return 1
profiles = {profile.id: profile for profile in connector_profiles_from_payload({"profiles": _profile_payloads()})}
if args.debug_smb:
_debug_smb(profiles["dev-smb"])
failures: list[str] = []
for profile_id, folder, file_path, expected in (
("dev-webdav", "GovOPlaN", "GovOPlaN/webdav-live.txt", "webdav live fixture"),
("dev-nextcloud", "GovOPlaN", "GovOPlaN/nextcloud-live.txt", "nextcloud live fixture"),
):
try:
_exercise_profile(profiles[profile_id], folder=folder, file_path=file_path, expected=expected)
except Exception as exc:
failures.append(f"{profile_id}: {exc}")
try:
_exercise_profile(profiles["dev-smb"], folder="GovOPlaN", file_path="GovOPlaN/smb-live.txt", expected="smb live fixture")
except Exception as exc:
message = f"dev-smb: {exc}"
if args.require_smb:
failures.append(message)
else:
print(f"SKIP {message}")
if failures:
for failure in failures:
print(f"FAIL {failure}")
return 1
print("OK connector dev stack smoke checks passed")
return 0
def _default_env() -> None:
defaults = {
"NEXTCLOUD_ADMIN_USER": "admin",
"NEXTCLOUD_ADMIN_PASSWORD": "govoplan-nextcloud-admin",
"WEBDAV_USER": "govoplan",
"WEBDAV_PASSWORD": "govoplan-webdav",
"SMB_SHARE_NAME": "files",
"SMB_USER": "govoplan",
"SMB_PASSWORD": "govoplan-smb",
}
for key, value in defaults.items():
os.environ.setdefault(key, value)
def _load_dotenv() -> None:
path = ROOT / ".env"
if not path.exists():
return
for line in path.read_text(encoding="utf-8").splitlines():
text = line.strip()
if not text or text.startswith("#") or "=" not in text:
continue
key, value = text.split("=", 1)
key = key.strip()
if not key or key in os.environ:
continue
os.environ[key] = value.strip().strip("\"'")
def _preflight_services(*, require_smb: bool) -> list[str]:
failures: list[str] = []
nextcloud_url = f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/status.php"
try:
response = httpx.get(nextcloud_url, timeout=3.0)
if response.status_code != 200:
failures.append(f"dev-nextcloud expected HTTP 200 at {nextcloud_url}, got HTTP {response.status_code}")
except httpx.HTTPError as exc:
failures.append(f"dev-nextcloud is not reachable at {nextcloud_url}: {exc}")
webdav_url = f"http://127.0.0.1:{os.getenv('WEBDAV_HOST_PORT', '9083')}/"
try:
response = httpx.get(webdav_url, timeout=3.0)
if response.status_code not in {200, 401}:
failures.append(f"dev-webdav expected HTTP 200/401 at {webdav_url}, got HTTP {response.status_code}")
except httpx.HTTPError as exc:
failures.append(f"dev-webdav is not reachable at {webdav_url}: {exc}")
if require_smb:
smb_port = int(os.getenv("SMB_HOST_PORT", "1445"))
try:
with socket.create_connection(("127.0.0.1", smb_port), timeout=3.0):
pass
except OSError as exc:
failures.append(f"dev-smb is not reachable at 127.0.0.1:{smb_port}: {exc}")
return failures
def _profile_payloads() -> list[dict[str, object]]:
return [
{
"id": "dev-webdav",
"provider": "webdav",
"endpoint_url": f"http://127.0.0.1:{os.getenv('WEBDAV_HOST_PORT', '9083')}/",
"credential_mode": "basic",
"username": os.getenv("WEBDAV_USER", "govoplan"),
"password_env": "WEBDAV_PASSWORD",
},
{
"id": "dev-nextcloud",
"provider": "nextcloud",
"endpoint_url": f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/remote.php/dav/files/{os.getenv('NEXTCLOUD_ADMIN_USER', 'admin')}/",
"credential_mode": "basic",
"username": os.getenv("NEXTCLOUD_ADMIN_USER", "admin"),
"password_env": "NEXTCLOUD_ADMIN_PASSWORD",
},
{
"id": "dev-smb",
"provider": "smb",
"endpoint_url": f"smb://127.0.0.1:{os.getenv('SMB_HOST_PORT', '1445')}/{os.getenv('SMB_SHARE_NAME', 'files')}",
"credential_mode": "basic",
"username": os.getenv("SMB_USER", "govoplan"),
"password_env": "SMB_PASSWORD",
},
]
def _exercise_profile(profile, *, folder: str, file_path: str, expected: str) -> None:
items = browse_connector_profile(profile, path=folder)
paths = {item.path for item in items}
if file_path not in paths:
raise RuntimeError(f"{file_path!r} not found; saw {sorted(paths)!r}")
downloaded = read_connector_file(profile, library_id="", path=file_path, max_bytes=1024 * 1024)
text = downloaded.data.decode("utf-8").strip()
if text != expected:
raise RuntimeError(f"{file_path!r} content mismatch: {text!r}")
print(f"OK {profile.id} browse/import {file_path}")
def _debug_smb(profile) -> None:
try:
import smbclient
except ImportError as exc:
print(f"DEBUG smbclient import failed: {exc}")
return
from govoplan_files.backend.storage.connector_browse import _smb_client_kwargs, _smb_location, _smb_unc_path
location = _smb_location(profile)
kwargs = _smb_client_kwargs(profile, location)
print(f"DEBUG SMB endpoint=//{location.server}:{location.port}/{location.share} root_path={location.root_path!r}")
print(f"DEBUG SMB user={kwargs.get('username')!r} require_signing={kwargs.get('require_signing')!r} auth_protocol={kwargs.get('auth_protocol')!r}")
try:
smbclient.reset_connection_cache()
except Exception as exc:
print(f"DEBUG SMB reset cache failed: {exc}")
for path in (_smb_unc_path(location, ""), _smb_unc_path(location, "GovOPlaN")):
try:
print(f"DEBUG SMB list {path}: {smbclient.listdir(path, **kwargs)}")
except Exception as exc:
print(f"DEBUG SMB list {path} failed: {type(exc).__name__}: {exc}")
def _seed_webdav_fixture() -> None:
path = ROOT / "data" / "webdav" / "GovOPlaN"
path.mkdir(parents=True, exist_ok=True)
(path / "webdav-live.txt").write_text("webdav live fixture\n", encoding="utf-8")
def _seed_smb_fixture() -> None:
path = ROOT / "data" / "smb" / "GovOPlaN"
path.mkdir(parents=True, exist_ok=True)
(path / "smb-live.txt").write_text("smb live fixture\n", encoding="utf-8")
def _seed_nextcloud_fixture() -> None:
base_url = f"http://127.0.0.1:{os.getenv('NEXTCLOUD_HOST_PORT', '9081')}/remote.php/dav/files/{os.getenv('NEXTCLOUD_ADMIN_USER', 'admin')}/GovOPlaN"
auth = (os.getenv("NEXTCLOUD_ADMIN_USER", "admin"), os.getenv("NEXTCLOUD_ADMIN_PASSWORD", "govoplan-nextcloud-admin"))
response = httpx.request("MKCOL", base_url, auth=auth, timeout=15.0)
if response.status_code not in {201, 405}:
raise RuntimeError(f"Nextcloud MKCOL failed with HTTP {response.status_code}: {response.text[:200]}")
response = httpx.put(f"{base_url}/nextcloud-live.txt", content=b"nextcloud live fixture\n", auth=auth, timeout=15.0)
if response.status_code not in {200, 201, 204}:
raise RuntimeError(f"Nextcloud PUT failed with HTTP {response.status_code}: {response.text[:200]}")
if __name__ == "__main__":
raise SystemExit(main())