GovOPlaN/govoplan-connectors
2
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

[Feature] Provide governed tabular datasource discovery and bounded preview #9

New Issue
Closed
opened 2026-07-28 00:58:09 +02:00 by zemion · 4 comments
zemion commented 2026-07-28 00:58:09 +02:00
Owner
Copy Link
Copy Source

Outcome

Provide the source-side contract needed by GovOPlaN/govoplan-dataflow#1 and the monthly data workflow.

Boundary

Connectors owns connection profiles, credentials references, health, source discovery, schema snapshots, query parameters, bounded sampling, and provider-specific pushdown. Dataflow owns transformation definitions and run lineage; Files owns stored artifacts.

Acceptance Criteria

  • A versioned capability lists visible tabular sources without exposing credentials.
  • Schema discovery returns stable source IDs, column names/types/nullability, discovery revision, and source fingerprint.
  • Bounded preview enforces row/byte/time limits and identifies whether data is live, cached, or file-backed.
  • Connector implementations declare supported filters/projections/aggregation pushdown; unsupported operations stay in Dataflow.
  • CSV/XLSX from Files and one SQL database adapter form the first tested providers.
  • Schema drift, stale credentials, unavailable sources, and tenant isolation have explicit diagnostics.

Related registry work: #7 and #6.

## Outcome Provide the source-side contract needed by https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/1 and the [monthly data workflow](https://git.add-ideas.de/GovOPlaN/govoplan/issues/8). ## Boundary Connectors owns connection profiles, credentials references, health, source discovery, schema snapshots, query parameters, bounded sampling, and provider-specific pushdown. Dataflow owns transformation definitions and run lineage; Files owns stored artifacts. ## Acceptance Criteria - [x] A versioned capability lists visible tabular sources without exposing credentials. - [x] Schema discovery returns stable source IDs, column names/types/nullability, discovery revision, and source fingerprint. - [ ] Bounded preview enforces row/byte/time limits and identifies whether data is live, cached, or file-backed. - [ ] Connector implementations declare supported filters/projections/aggregation pushdown; unsupported operations stay in Dataflow. - [ ] CSV/XLSX from Files and one SQL database adapter form the first tested providers. - [ ] Schema drift, stale credentials, unavailable sources, and tenant isolation have explicit diagnostics. Related registry work: https://git.add-ideas.de/GovOPlaN/govoplan-connectors/issues/7 and https://git.add-ideas.de/GovOPlaN/govoplan-connectors/issues/6.
zemion commented 2026-07-28 11:18:45 +02:00
Author
Owner
Copy Link
Copy Source

Codex State: progress

Summary

  • Converted Connectors from a documentation-only shell into an installable module with migrations, permissions, role templates, APIs, uninstall guards, and versioned tabular capabilities (ba5ccea).
  • The first tenant-isolated provider imports bounded immutable JSON/CSV snapshots, infers schemas, fingerprints content, rejects stale reads, and returns opaque source references without credentials.
  • CSV parsing handles BOMs, delimiters, malformed rows, scalar inference, and leading-zero identifiers through the shared core contract.

Verification

  • python -m unittest discover -s govoplan-connectors/tests (7 passed)
  • python -m unittest discover -s govoplan-dataflow/tests (24 passed)
  • tools/checks/check-contracts.sh (45 contracts passed)
  • tools/checks/check-focused.sh (core, module, API, and WebUI checks passed)
  • Composed JSON/CSV snapshot and connector-backed preview probe passed.

Next / Blocked

  • Implement Files-backed CSV/XLSX discovery, one SQL database adapter, explicit pushdown declarations, source-health/time diagnostics, and stale-credential behavior before closing.
## Codex State: progress ### Summary - Converted Connectors from a documentation-only shell into an installable module with migrations, permissions, role templates, APIs, uninstall guards, and versioned tabular capabilities (`ba5ccea`). - The first tenant-isolated provider imports bounded immutable JSON/CSV snapshots, infers schemas, fingerprints content, rejects stale reads, and returns opaque source references without credentials. - CSV parsing handles BOMs, delimiters, malformed rows, scalar inference, and leading-zero identifiers through the shared core contract. ### Verification - `python -m unittest discover -s govoplan-connectors/tests` (7 passed) - `python -m unittest discover -s govoplan-dataflow/tests` (24 passed) - `tools/checks/check-contracts.sh` (45 contracts passed) - `tools/checks/check-focused.sh` (core, module, API, and WebUI checks passed) - Composed JSON/CSV snapshot and connector-backed preview probe passed. ### Next / Blocked - Implement Files-backed CSV/XLSX discovery, one SQL database adapter, explicit pushdown declarations, source-health/time diagnostics, and stale-credential behavior before closing.
zemion added
status
in-progress
 and removed
status
ready
 labels 2026-07-28 11:19:31 +02:00
zemion commented 2026-07-28 12:48:21 +02:00
Author
Owner
Copy Link
Copy Source

Codex State: progress

Implemented connectors.datasourceOrigins@0.1.0 and a capability adapter over
the existing bounded tabular snapshot provider in 27302f0. Datasources now
distinguishes whether a registered origin is live or cached and owns
frozen/materialized state; Connectors remains responsible for source kind,
provider health, credentials, and pushdown.

Validated tenant isolation, schema/fingerprint metadata, bounded reads, error
translation, and the composed Connector -> Datasources -> Dataflow path. SQL,
REST, directory, and managed-file providers plus explicit pushdown/health
diagnostics remain open.

## Codex State: progress Implemented `connectors.datasourceOrigins@0.1.0` and a capability adapter over the existing bounded tabular snapshot provider in `27302f0`. Datasources now distinguishes whether a registered origin is live or cached and owns frozen/materialized state; Connectors remains responsible for source kind, provider health, credentials, and pushdown. Validated tenant isolation, schema/fingerprint metadata, bounded reads, error translation, and the composed Connector -> Datasources -> Dataflow path. SQL, REST, directory, and managed-file providers plus explicit pushdown/health diagnostics remain open.
zemion commented 2026-08-04 12:19:36 +02:00
Author
Owner
Copy Link
Copy Source

Codex State: progress

Summary

  • Implemented provider-enforced preview ceilings for rows, exact serialized bytes, and elapsed time in commit 20146ef.
  • Source discovery now declares live/cached/file-backed/static mode, structured health, and projection/filter/aggregation/sort/pagination pushdown.
  • Connector and Datasource APIs expose effective limits and redacted diagnostics; unavailable sources are distinct from invalid requests.

Verification

  • 29 Connectors tests passed; focused cross-repository check and 50 WebUI permutations passed.

Next / Blocked

  • Remaining: managed Files CSV/XLSX and one SQL origin adapter, plus live credential/drift/unavailability diagnostics for those providers.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Implemented provider-enforced preview ceilings for rows, exact serialized bytes, and elapsed time in commit 20146ef. - Source discovery now declares live/cached/file-backed/static mode, structured health, and projection/filter/aggregation/sort/pagination pushdown. - Connector and Datasource APIs expose effective limits and redacted diagnostics; unavailable sources are distinct from invalid requests. ### Verification - `29 Connectors tests passed; focused cross-repository check and 50 WebUI permutations passed.` ### Next / Blocked - Remaining: managed Files CSV/XLSX and one SQL origin adapter, plus live credential/drift/unavailability diagnostics for those providers. Suggested status label: `status/in-progress`
zemion commented 2026-08-21 19:57:03 +02:00
Author
Owner
Copy Link
Copy Source

Codex State: completed

Delivered

  • Core 142ccbc / v0.1.19 defines the optional files.tabular_content@1.0.0 exact-version contract; f75ad48 also removes the module-entitlement test-order dependency found by the full suite.
  • Files a8c3192 / v0.1.19 supplies authorized CSV/XLSX discovery and exact-version content reads through Files access, integrity, quarantine, encryption, retention/hold, and audit behavior.
  • Connectors 2c2b11f / v0.1.20 adds persisted managed-file and PostgreSQL origins, explicit reviewed refresh, schema/configuration/credential drift blocking, redacted health failures, projection/pagination pushdown, read-only SQL transactions, connection/statement limits, and bounded CSV/XLSX parsing.
  • Root 7f2f896 synchronizes package releases and classifies the three capability-first endpoints.

Acceptance evidence

  • Row, exact serialized-byte, elapsed-time, file/archive/expansion, SQL connection, and SQL statement ceilings are enforced.
  • Source mode and structured health distinguish live PostgreSQL, file-backed exact versions, and cached immutable snapshots.
  • Unsupported filtering, aggregation, and sorting remain undeclared for Dataflow; projection and pagination are declared and pushed down.
  • Tests cover CSV, XLSX worksheet selection and typing, SQL discovery/projection, tenant isolation, inactive/missing/rotated credentials, connector configuration drift, schema drift, unavailable Files capability, unsafe endpoint credentials, explicit file-version adoption, and audit-safe errors.

Verification

  • Files: 163 tests and 34 subtests passed.
  • Connectors: 49 tests and 3 subtests passed.
  • Manifest shapes and strict platform inventory passed for 68 modules, 153 providers, and 95 requirements.
  • tools/checks/check-focused.sh passed all backend/composition checks, 59 WebUI module permutations, the full-product bundle budgets, and 8 Playwright conformance journeys.

All acceptance criteria are satisfied; closing.

## Codex State: completed ### Delivered - Core `142ccbc` / `v0.1.19` defines the optional `files.tabular_content@1.0.0` exact-version contract; `f75ad48` also removes the module-entitlement test-order dependency found by the full suite. - Files `a8c3192` / `v0.1.19` supplies authorized CSV/XLSX discovery and exact-version content reads through Files access, integrity, quarantine, encryption, retention/hold, and audit behavior. - Connectors `2c2b11f` / `v0.1.20` adds persisted managed-file and PostgreSQL origins, explicit reviewed refresh, schema/configuration/credential drift blocking, redacted health failures, projection/pagination pushdown, read-only SQL transactions, connection/statement limits, and bounded CSV/XLSX parsing. - Root `7f2f896` synchronizes package releases and classifies the three capability-first endpoints. ### Acceptance evidence - Row, exact serialized-byte, elapsed-time, file/archive/expansion, SQL connection, and SQL statement ceilings are enforced. - Source mode and structured health distinguish live PostgreSQL, file-backed exact versions, and cached immutable snapshots. - Unsupported filtering, aggregation, and sorting remain undeclared for Dataflow; projection and pagination are declared and pushed down. - Tests cover CSV, XLSX worksheet selection and typing, SQL discovery/projection, tenant isolation, inactive/missing/rotated credentials, connector configuration drift, schema drift, unavailable Files capability, unsafe endpoint credentials, explicit file-version adoption, and audit-safe errors. ### Verification - Files: 163 tests and 34 subtests passed. - Connectors: 49 tests and 3 subtests passed. - Manifest shapes and strict platform inventory passed for 68 modules, 153 providers, and 95 requirements. - `tools/checks/check-focused.sh` passed all backend/composition checks, 59 WebUI module permutations, the full-product bundle budgets, and 8 Playwright conformance journeys. All acceptance criteria are satisfied; closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/api
HTTP API contracts, routers, schemas, or API smoke behavior.
 area/auth
Authentication, sessions, access bootstrap, or login behavior.
 area/db
Database sessions, models, transactions, or persistence primitives.
 area/devex
Local developer workflow, scripts, tests, tooling, or release helpers.
 area/docs
Durable documentation and project guidance.
 area/governance
Governance policy, audit, privacy, retention, or compliance behavior.
 area/marketing
Public website, product messaging, publication copy, or legal page content.
 area/migrations
Alembic migrations, schema bootstrap, or persistence evolution.
 area/module-system
Module discovery, manifests, capabilities, routing, or optional integrations.
 area/rbac
Permissions, roles, delegation, or authorization policy.
 area/release
Versioning, release locks, tags, packaging, or dependency pins.
 area/security
Security posture, static analysis, supply-chain hardening, or vulnerability remediation.
 area/tenancy
Tenant boundaries, provisioning, or tenant-scoped data behavior.
 area/webui
Shared WebUI shell, frontend components, routing, or frontend tests.
 audit/complexity
Complexity finding from Radon, Xenon, or equivalent maintainability scans.
 audit/duplication
Duplicated-code finding from jscpd or equivalent similarity scans.
 audit/false-positive
Audit finding reviewed as a narrow false positive or acceptable risk.
 audit/needs-design
Audit finding that needs an architectural or product decision before implementation.
 audit/quick-fix
Audit finding that appears narrow and directly fixable.
 audit/structural
Audit finding that needs design, refactoring, or behavior review.
 codex/needs-human
Needs an explicit human decision before Codex should implement.
 codex/ready
Suitable for Codex to pick up with the existing issue context.
 module/access
GovOPlaN access, identity, authentication, RBAC, and administration behavior.
 module/addresses
GovOPlaN Addresses module behavior or integration.
 module/admin
GovOPlaN Admin module behavior or integration.
 module/appointments
GovOPlaN Appointments module behavior or integration.
 module/approvals
GovOPlaN Approvals module behavior or integration.
 module/audit
GovOPlaN Audit module behavior or integration.
 module/calendar
GovOPlaN Calendar module behavior or integration.
 module/campaign
GovOPlaN campaign module behavior or integration.
 module/cases
GovOPlaN Cases module behavior or integration.
 module/committee
GovOPlaN Committee module behavior or integration.
 module/connectors
GovOPlaN Connectors module behavior or integration.
 module/core
GovOPlaN core runner, shared primitives, shell, or extension points.
 module/dashboard
GovOPlaN Dashboard module behavior or integration.
 module/dataflow
GovOPlaN Dataflow module behavior or integration.
 module/datasources
GovOPlaN governed datasource contracts, catalogs, and integrations.
 module/decisions
GovOPlaN formal Decisions module behavior or integration.
 module/dist-lists
GovOPlaN Distribution Lists module behavior or integration.
 module/dms
GovOPlaN Dms module behavior or integration.
 module/docs
GovOPlaN Docs module behavior or integration.
 module/encryption
GovOPlaN Encryption key custody, cryptographic policy, and E2EE integration.
 module/erp
GovOPlaN Erp module behavior or integration.
 module/evaluation
GovOPlaN Evaluation module behavior or integration.
 module/files
GovOPlaN files module behavior or integration.
 module/fit-connect
GovOPlaN Fit Connect module behavior or integration.
 module/forms
GovOPlaN Forms module behavior or integration.
 module/forms-runtime
GovOPlaN Forms Runtime module behavior or integration.
 module/helpdesk
GovOPlaN Helpdesk module behavior or integration.
 module/identity
GovOPlaN Identity module behavior or integration.
 module/identity-trust
GovOPlaN Identity Trust module behavior or integration.
 module/idm
GovOPlaN Idm module behavior or integration.
 module/ledger
GovOPlaN Ledger module behavior or integration.
 module/mail
GovOPlaN mail module behavior or integration.
 module/mandates
GovOPlaN Mandates, jurisdiction, responsibility, and authority behavior or integration.
 module/notifications
GovOPlaN Notifications module behavior or integration.
 module/ops
GovOPlaN Ops module behavior or integration.
 module/organizations
GovOPlaN Organizations module behavior or integration.
 module/parties
GovOPlaN procedure Parties, representation, and delivery-authority behavior or integration.
 module/payments
GovOPlaN Payments module behavior or integration.
 module/permits
GovOPlaN Permits module behavior or integration.
 module/policy
GovOPlaN Policy module behavior or integration.
 module/poll
GovOPlaN Poll module behavior or integration.
 module/portal
GovOPlaN Portal module behavior or integration.
 module/postbox
GovOPlaN Postbox module behavior or integration.
 module/projects
GovOPlaN Projects module behavior or integration.
 module/quick-access
GovOPlaN configurable task-local Quick Access behavior and integrations.
 module/records
GovOPlaN Records and eAkte lifecycle behavior or integration.
 module/reporting
GovOPlaN Reporting module behavior or integration.
 module/risk-compliance
GovOPlaN Risk Compliance module behavior or integration.
 module/scheduling
GovOPlaN Scheduling module behavior or integration.
 module/search
GovOPlaN Search module behavior or integration.
 module/services
GovOPlaN versioned institutional Services behavior or integration.
 module/tasks
GovOPlaN Tasks module behavior or integration.
 module/templates
GovOPlaN Templates module behavior or integration.
 module/tenancy
GovOPlaN Tenancy module behavior or integration.
 module/tickets
GovOPlaN Tickets module behavior or integration.
 module/views
GovOPlaN governed task views, interface projections, and workflow view integration.
 module/voting
GovOPlaN Voting module behavior or integration.
 module/wiki
GovOPlaN Wiki module behavior or integration.
 module/workflow
GovOPlaN Workflow module behavior or integration.
 module/workflow-engine
GovOPlaN Workflow Engine runtime, persistence, or integration.
 module/xoev
GovOPlaN Xoev module behavior or integration.
 module/xrechnung
GovOPlaN Xrechnung module behavior or integration.
 module/xta-osci
GovOPlaN Xta Osci module behavior or integration.
priority
p0
Immediate stop-the-line priority.
priority
p1
High priority for the next focused work window.
priority
p2
Normal planned priority.
priority
p3
Low priority or opportunistic cleanup.
source/backlog-import
Imported from markdown backlog, roadmap, plan, or TODO files.
 source/security-audit
Created from a structured security or code-quality audit report.
 source/todo-scan
Imported from inline TODO/FIXME/HACK markers by the Gitea TODO importer.
status
blocked
Cannot progress without a decision, dependency, credential, or external change.
status
in-progress
Currently being worked.
status
needs-info
Needs clarifying input before implementation can proceed safely.
status
ready
Ready for implementation.
status
triage
Needs review, ownership, priority, or acceptance criteria.
type
bug
A reproducible defect, regression, or incorrect behavior.
type
debt
Cleanup, refactoring, risk reduction, or deferred engineering work.
type
docs
Documentation, process, or developer workflow work.
type
feature
New user-visible behavior or platform capability.
type
task
Implementation, maintenance, migration, or operational work.
type
user-story
End-to-end user journey or real-world process story used to steer product slices.
No labels area/api area/security codex/ready module/connectors module/dataflow module/datasources
priority
p1
status
in-progress
type
feature
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
govoplan-package-publisher zemion (Albrecht Degering)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-connectors#9