add-ideas/govoplan-files
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

[Feature] Add S3-compatible object-storage connector for external file providers #20

New Issue
Closed
opened 2026-07-09 12:40:27 +02:00 by zemion · 1 comment
zemion commented 2026-07-09 12:40:27 +02:00
Owner
Copy Link

Objective

Implement an S3-compatible object-storage connector for govoplan-files as an external file provider.

This is not the managed-file blob storage backend. It is a connector profile that lets users/admins browse an external S3-compatible bucket or prefix, select objects, and import/freeze them as managed GovOPlaN files with provenance.

Scope

  • Owning repository: govoplan-files.
  • Provider key: s3 or s3-compatible.
  • Support AWS S3-compatible APIs, with MinIO as the development/test target.
  • Fit into the existing connector profile, credential, policy, linked-space, browse/import, sync, and audit model.

Expected behavior

  • Add provider descriptor for S3-compatible object storage.
  • Support endpoint URL, region, bucket, base prefix, path-style/virtual-host style option where needed, and TLS verification configuration where safe.
  • Support secret references or encrypted stored credentials for access key / secret key; do not expose secret values in API responses.
  • Add profile test/health check that validates bucket/prefix access without leaking object metadata unnecessarily.
  • Browse prefixes and objects with pagination/continuation tokens.
  • Import one selected object into managed storage, preserving provider, bucket, key, version/ETag where available, checksum/size/MIME type where available, and imported file reference.
  • Enforce connector policy before profile use, browse, import, linked-space creation, and sync.
  • Emit connector audit events for test, browse/access, import, and denial paths.
  • Add a Docker/MinIO smoke fixture comparable to the existing WebDAV/Nextcloud/SMB connector test bed.

Acceptance criteria

  • S3 provider appears in /api/v1/files/connectors/providers with implemented capability metadata.
  • Admin/settings UI can create or select an S3 connector profile and credential profile.
  • Files UI can browse a bucket/prefix and import an object into managed storage.
  • Imported files include enough provenance to explain source bucket/key/version/ETag later.
  • Existing connector policy checks apply consistently to S3 provider, profile, credential, and path/prefix.
  • Tests cover provider descriptor, credential redaction, profile test, list pagination, import provenance, policy denial, and unavailable dependency behavior.
  • Dev smoke can run against a local MinIO container without real cloud credentials.

Notes

govoplan-connectors should keep the catalogue/inventory entry shape, but executable file-provider behavior belongs in govoplan-files unless S3 grows into remote-write, lifecycle administration, background indexing, or archive-management behavior beyond on-demand browse/import.

## Objective Implement an S3-compatible object-storage connector for `govoplan-files` as an external file provider. This is **not** the managed-file blob storage backend. It is a connector profile that lets users/admins browse an external S3-compatible bucket or prefix, select objects, and import/freeze them as managed GovOPlaN files with provenance. ## Scope - Owning repository: `govoplan-files`. - Provider key: `s3` or `s3-compatible`. - Support AWS S3-compatible APIs, with MinIO as the development/test target. - Fit into the existing connector profile, credential, policy, linked-space, browse/import, sync, and audit model. ## Expected behavior - Add provider descriptor for S3-compatible object storage. - Support endpoint URL, region, bucket, base prefix, path-style/virtual-host style option where needed, and TLS verification configuration where safe. - Support secret references or encrypted stored credentials for access key / secret key; do not expose secret values in API responses. - Add profile test/health check that validates bucket/prefix access without leaking object metadata unnecessarily. - Browse prefixes and objects with pagination/continuation tokens. - Import one selected object into managed storage, preserving provider, bucket, key, version/ETag where available, checksum/size/MIME type where available, and imported file reference. - Enforce connector policy before profile use, browse, import, linked-space creation, and sync. - Emit connector audit events for test, browse/access, import, and denial paths. - Add a Docker/MinIO smoke fixture comparable to the existing WebDAV/Nextcloud/SMB connector test bed. ## Acceptance criteria - S3 provider appears in `/api/v1/files/connectors/providers` with implemented capability metadata. - Admin/settings UI can create or select an S3 connector profile and credential profile. - Files UI can browse a bucket/prefix and import an object into managed storage. - Imported files include enough provenance to explain source bucket/key/version/ETag later. - Existing connector policy checks apply consistently to S3 provider, profile, credential, and path/prefix. - Tests cover provider descriptor, credential redaction, profile test, list pagination, import provenance, policy denial, and unavailable dependency behavior. - Dev smoke can run against a local MinIO container without real cloud credentials. ## Notes `govoplan-connectors` should keep the catalogue/inventory entry shape, but executable file-provider behavior belongs in `govoplan-files` unless S3 grows into remote-write, lifecycle administration, background indexing, or archive-management behavior beyond on-demand browse/import.
zemion added the
type
feature
status
ready
priority
p2
area/api
module/files
codex/ready
 labels 2026-07-09 12:40:27 +02:00
zemion commented 2026-07-12 01:12:18 +02:00
Author
Owner
Copy Link

Codex State: done

Summary

  • Added S3-compatible provider descriptor, backend validation, profile/UI provider keys, and optional govoplan-files[s3] dependency.
  • Implemented S3 bucket listing, prefix/object browsing, continuation-token plumbing, and object import/sync download path with bucket/key/version/ETag/checksum provenance.
  • Added MinIO to the local connector compose stack and optional S3 smoke support.

Changed Files

  • pyproject.toml
  • src/govoplan_files/backend/storage/connector_providers.py
  • src/govoplan_files/backend/storage/connector_profiles.py
  • src/govoplan_files/backend/storage/connector_browse.py
  • src/govoplan_files/backend/storage/connector_imports.py
  • src/govoplan_files/backend/schemas.py
  • src/govoplan_files/backend/router.py
  • webui/src/api/files.ts
  • webui/src/features/files/FileConnectorSettingsPanel.tsx
  • dev/connectors/docker-compose.yml
  • dev/connectors/smoke.py
  • dev/connectors/README.md
  • tests/test_connector_providers.py

Verification

  • PYTHONPATH=/mnt/DATA/git/govoplan-core/src:/mnt/DATA/git/govoplan-files/src /mnt/DATA/git/govoplan/.venv/bin/python -m unittest tests.test_connector_providers tests.test_transfer_helpers tests.test_router_contract tests.test_access_provider
  • PYTHONPATH=/mnt/DATA/git/govoplan-core/src:/mnt/DATA/git/govoplan-files/src /mnt/DATA/git/govoplan/.venv/bin/python -m compileall -q src/govoplan_files tests dev/connectors/smoke.py
  • git diff --check
## Codex State: done ### Summary - Added S3-compatible provider descriptor, backend validation, profile/UI provider keys, and optional `govoplan-files[s3]` dependency. - Implemented S3 bucket listing, prefix/object browsing, continuation-token plumbing, and object import/sync download path with bucket/key/version/ETag/checksum provenance. - Added MinIO to the local connector compose stack and optional S3 smoke support. ### Changed Files - `pyproject.toml` - `src/govoplan_files/backend/storage/connector_providers.py` - `src/govoplan_files/backend/storage/connector_profiles.py` - `src/govoplan_files/backend/storage/connector_browse.py` - `src/govoplan_files/backend/storage/connector_imports.py` - `src/govoplan_files/backend/schemas.py` - `src/govoplan_files/backend/router.py` - `webui/src/api/files.ts` - `webui/src/features/files/FileConnectorSettingsPanel.tsx` - `dev/connectors/docker-compose.yml` - `dev/connectors/smoke.py` - `dev/connectors/README.md` - `tests/test_connector_providers.py` ### Verification - `PYTHONPATH=/mnt/DATA/git/govoplan-core/src:/mnt/DATA/git/govoplan-files/src /mnt/DATA/git/govoplan/.venv/bin/python -m unittest tests.test_connector_providers tests.test_transfer_helpers tests.test_router_contract tests.test_access_provider` - `PYTHONPATH=/mnt/DATA/git/govoplan-core/src:/mnt/DATA/git/govoplan-files/src /mnt/DATA/git/govoplan/.venv/bin/python -m compileall -q src/govoplan_files tests dev/connectors/smoke.py` - `git diff --check`
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/api
HTTP API contracts, routers, schemas, or API smoke behavior.
area/auth
Authentication, sessions, access bootstrap, or login behavior.
area/db
Database sessions, models, transactions, or persistence primitives.
area/deployment
Deployment, release, operations, or runtime packaging.
area/devex
Local developer workflow, scripts, tests, tooling, or release helpers.
area/docs
Durable documentation and project guidance.
area/governance
Governance policy, audit, privacy, retention, or compliance behavior.
area/marketing
Public website, product messaging, publication copy, or legal page content.
area/migrations
Alembic migrations, schema bootstrap, or persistence evolution.
area/module-system
Module discovery, manifests, capabilities, routing, or optional integrations.
area/rbac
Permissions, roles, delegation, or authorization policy.
area/release
Versioning, release locks, tags, packaging, or dependency pins.
area/security
Security posture, static analysis, supply-chain hardening, or vulnerability remediation.
area/tenancy
Tenant boundaries, provisioning, or tenant-scoped data behavior.
area/webui
Shared WebUI shell, frontend components, routing, or frontend tests.
audit/complexity
Complexity finding from Radon, Xenon, or equivalent maintainability scans.
audit/duplication
Duplicated-code finding from jscpd or equivalent similarity scans.
audit/false-positive
Audit finding reviewed as a narrow false positive or acceptable risk.
audit/needs-design
Audit finding that needs an architectural or product decision before implementation.
audit/quick-fix
Audit finding that appears narrow and directly fixable.
audit/structural
Audit finding that needs design, refactoring, or behavior review.
codex/needs-human
Needs an explicit human decision before Codex should implement.
codex/ready
Suitable for Codex to pick up with the existing issue context.
module/access
GovOPlaN access, identity, authentication, RBAC, and administration behavior.
module/addresses
GovOPlaN Addresses module behavior or integration.
module/admin
GovOPlaN Admin module behavior or integration.
module/appointments
GovOPlaN Appointments module behavior or integration.
module/approvals
GovOPlaN Approvals module behavior or integration.
module/assets
GovOPlaN Assets module behavior or integration.
module/audit
GovOPlaN Audit module behavior or integration.
module/booking
GovOPlaN Booking module behavior or integration.
module/calendar
GovOPlaN Calendar module behavior or integration.
module/campaign
GovOPlaN campaign module behavior or integration.
module/cases
GovOPlaN Cases module behavior or integration.
module/certificates
GovOPlaN Certificates module behavior or integration.
module/committee
GovOPlaN Committee module behavior or integration.
module/connectors
GovOPlaN Connectors module behavior or integration.
module/consultation
GovOPlaN Consultation module behavior or integration.
module/contracts
GovOPlaN Contracts module behavior or integration.
module/core
GovOPlaN core runner, shared primitives, shell, or extension points.
module/dist-lists
GovOPlaN Distribution Lists module behavior or integration.
module/dms
GovOPlaN Dms module behavior or integration.
module/docs
GovOPlaN documentation layer behavior or integration.
module/erp
GovOPlaN Erp module behavior or integration.
module/evaluation
GovOPlaN Evaluation module behavior or integration.
module/facilities
GovOPlaN Facilities module behavior or integration.
module/files
GovOPlaN files module behavior or integration.
module/fit-connect
GovOPlaN Fit Connect module behavior or integration.
module/forms
GovOPlaN Forms module behavior or integration.
module/forms-runtime
GovOPlaN Forms Runtime module behavior or integration.
module/grants
GovOPlaN Grants module behavior or integration.
module/helpdesk
GovOPlaN Helpdesk module behavior or integration.
module/identity
GovOPlaN Identity module behavior or integration.
module/identity-trust
GovOPlaN Identity Trust module behavior or integration.
module/idm
GovOPlaN Idm module behavior or integration.
module/inspections
GovOPlaN Inspections module behavior or integration.
module/issue-reporting
GovOPlaN Issue Reporting module behavior or integration.
module/learning
GovOPlaN Learning module behavior or integration.
module/ledger
GovOPlaN Ledger module behavior or integration.
module/mail
GovOPlaN mail module behavior or integration.
module/notifications
GovOPlaN Notifications module behavior or integration.
module/ops
GovOPlaN Ops module behavior or integration.
module/organizations
GovOPlaN Organizations module behavior or integration.
module/payments
GovOPlaN Payments module behavior or integration.
module/permits
GovOPlaN Permits module behavior or integration.
module/policy
GovOPlaN Policy module behavior or integration.
module/poll
GovOPlaN Poll module behavior or integration.
module/portal
GovOPlaN Portal module behavior or integration.
module/postbox
GovOPlaN Postbox module behavior or integration.
module/procurement
GovOPlaN Procurement module behavior or integration.
module/records
GovOPlaN Records module behavior or integration.
module/reporting
GovOPlaN Reporting module behavior or integration.
module/resources
GovOPlaN Resources module behavior or integration.
module/risk-compliance
GovOPlaN Risk Compliance module behavior or integration.
module/scheduling
GovOPlaN Scheduling module behavior or integration.
module/search
GovOPlaN Search module behavior or integration.
module/tasks
GovOPlaN Tasks module behavior or integration.
module/templates
GovOPlaN Templates module behavior or integration.
module/tenancy
GovOPlaN Tenancy module behavior or integration.
module/transparency
GovOPlaN Transparency module behavior or integration.
module/web
GovOPlaN public website, product page, or publication content.
module/workflow
GovOPlaN Workflow module behavior or integration.
module/xoev
GovOPlaN Xoev module behavior or integration.
module/xrechnung
GovOPlaN Xrechnung module behavior or integration.
module/xta-osci
GovOPlaN Xta Osci module behavior or integration.
priority
p0
Immediate stop-the-line priority.
priority
p1
High priority for the next focused work window.
priority
p2
Normal planned priority.
priority
p3
Low priority or opportunistic cleanup.
source/backlog-import
Imported from markdown backlog, roadmap, plan, or TODO files.
source/module-roadmap
Created from GovOPlaN module and integration roadmap planning.
source/security-audit
Created from a structured security or code-quality audit report.
source/todo-scan
Imported from inline TODO/FIXME/HACK markers by the Gitea TODO importer.
status
blocked
Cannot progress without a decision, dependency, credential, or external change.
status
in-progress
Currently being worked.
status
needs-info
Needs clarifying input before implementation can proceed safely.
status
ready
Ready for implementation.
status
triage
Needs review, ownership, priority, or acceptance criteria.
type
bug
A reproducible defect, regression, or incorrect behavior.
type
debt
Cleanup, refactoring, risk reduction, or deferred engineering work.
type
docs
Documentation, process, or developer workflow work.
type
feature
New user-visible behavior or platform capability.
type
task
Implementation, maintenance, migration, or operational work.
type
user-story
End-to-end user journey or real-world process story used to steer product slices.
No Label
area/api
codex/ready
module/files
priority
p2
status
ready
type
feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
zemion
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: add-ideas/govoplan-files#20
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?