GovOPlaN/govoplan-datasources
2
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

[Epic] Deliver the governed datasource catalogue and lifecycle #1

New Issue
Closed
opened 2026-07-28 12:48:18 +02:00 by zemion · 3 comments
zemion commented 2026-07-28 12:48:18 +02:00
Owner
Copy Link
Copy Source

Outcome

Provide one provider-neutral, tenant-governed data layer that Dataflow,
Workflow, Reporting, Risk Compliance, and domain modules can consume without
importing connector implementations or retaining credentials.

Boundary

  • Datasources owns source identity, visibility, live/cached/static mode,
    staging, immutable materializations, frozen states, schema/fingerprints,
    provenance, and bounded reads.
  • Connectors owns external protocols, endpoints, credentials, discovery,
    source-side filtering/pushdown, and provider health.
  • Dataflow owns transformations and lineage.
  • Workflow owns orchestration, approvals, retries, and module handoffs.

First Slice

  • Package, module manifest, permissions, migration, uninstall guard, and WebUI route.
  • Tenant-isolated live, cached, and static datasource catalogue.
  • Bounded JSON/CSV staging and promotion.
  • Append-only materializations, current state, and explicit frozen states.
  • Schema versions, content fingerprints, provenance, and bounded preview.
  • Connector origin adapter and Dataflow capability consumer.
  • Focused composition test from Connector through Datasources to Dataflow.
  • Production storage and concurrency hardening.
  • Quality, schema compatibility, approval, retention, and policy gates.
  • Module-output publication and Workflow lifecycle integration.

Architecture: https://git.add-ideas.de/GovOPlaN/govoplan/src/branch/main/docs/DATASOURCE_AND_DEFINITION_GRAPH_ARCHITECTURE.md

Architecture reconciliation (2026-07-31)

The implemented lifecycle remains the foundation. Typed institutional catalogue metadata and dependency impact are the child slice GovOPlaN/govoplan-datasources#6. This does not move connector profiles/credentials, transformations, or report execution into Datasources.

## Outcome Provide one provider-neutral, tenant-governed data layer that Dataflow, Workflow, Reporting, Risk Compliance, and domain modules can consume without importing connector implementations or retaining credentials. ## Boundary - Datasources owns source identity, visibility, live/cached/static mode, staging, immutable materializations, frozen states, schema/fingerprints, provenance, and bounded reads. - Connectors owns external protocols, endpoints, credentials, discovery, source-side filtering/pushdown, and provider health. - Dataflow owns transformations and lineage. - Workflow owns orchestration, approvals, retries, and module handoffs. ## First Slice - [x] Package, module manifest, permissions, migration, uninstall guard, and WebUI route. - [x] Tenant-isolated live, cached, and static datasource catalogue. - [x] Bounded JSON/CSV staging and promotion. - [x] Append-only materializations, current state, and explicit frozen states. - [x] Schema versions, content fingerprints, provenance, and bounded preview. - [x] Connector origin adapter and Dataflow capability consumer. - [x] Focused composition test from Connector through Datasources to Dataflow. - [x] Production storage and concurrency hardening. - [x] Quality, schema compatibility, approval, retention, and policy gates. - [x] Module-output publication and Workflow lifecycle integration. Architecture: https://git.add-ideas.de/GovOPlaN/govoplan/src/branch/main/docs/DATASOURCE_AND_DEFINITION_GRAPH_ARCHITECTURE.md ## Architecture reconciliation (2026-07-31) The implemented lifecycle remains the foundation. Typed institutional catalogue metadata and dependency impact are the child slice [GovOPlaN/govoplan-datasources#6](https://git.add-ideas.de/GovOPlaN/govoplan-datasources/issues/6). This does not move connector profiles/credentials, transformations, or report execution into Datasources. <!-- codex-institutional-governance-datasources-2026-07-31 -->
zemion commented 2026-08-06 16:17:46 +02:00
Author
Owner
Copy Link
Copy Source

Codex State: progress

Pushed 492306f: Datasources now contributes a permission-aware native PostgreSQL Search source. Backfill is tenant-bounded and current/non-deleted, emits only safe catalogue metadata, excludes rows/schema/provider references/credentials/provenance, rechecks current catalogue authorization, and deep-links into the selected datasource.

All 31 Datasources tests pass (two optional PostgreSQL integration cases skipped without the test URL), with Ruff and WebUI typechecking green. The epic remains open for its production hardening, quality/policy gates, and module-output/Workflow lifecycle criteria.

## Codex State: progress Pushed `492306f`: Datasources now contributes a permission-aware native PostgreSQL Search source. Backfill is tenant-bounded and current/non-deleted, emits only safe catalogue metadata, excludes rows/schema/provider references/credentials/provenance, rechecks current catalogue authorization, and deep-links into the selected datasource. All 31 Datasources tests pass (two optional PostgreSQL integration cases skipped without the test URL), with Ruff and WebUI typechecking green. The epic remains open for its production hardening, quality/policy gates, and module-output/Workflow lifecycle criteria.
zemion commented 2026-08-22 20:01:42 +02:00
Author
Owner
Copy Link
Copy Source

Progress: child #2 is complete and released in v0.1.21. Datasources now covers quality/schema gates, optional attributable approval for stages and cached refreshes, explicit preview/apply retention for stages/materializations/frozen evidence, and hash-chained lifecycle evidence. Full focused workspace verification passed. Parent remains open for its remaining child slices.

Progress: child #2 is complete and released in v0.1.21. Datasources now covers quality/schema gates, optional attributable approval for stages and cached refreshes, explicit preview/apply retention for stages/materializations/frozen evidence, and hash-chained lifecycle evidence. Full focused workspace verification passed. Parent remains open for its remaining child slices.
zemion commented 2026-08-22 20:02:28 +02:00
Author
Owner
Copy Link
Copy Source

Epic completion verified after #2 closed. All child slices #2-#8 are complete: lifecycle quality/approval/retention, immutable output publication and terminal handoffs, concurrency/storage hardening, policy-aware visibility, institutional catalogue semantics, interface-pattern migration, and DSAR coverage. The Connector -> Datasources -> pinned Dataflow composition and full focused workspace gate pass on released Datasources v0.1.21.

Epic completion verified after #2 closed. All child slices #2-#8 are complete: lifecycle quality/approval/retention, immutable output publication and terminal handoffs, concurrency/storage hardening, policy-aware visibility, institutional catalogue semantics, interface-pattern migration, and DSAR coverage. The Connector -> Datasources -> pinned Dataflow composition and full focused workspace gate pass on released Datasources v0.1.21.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/api
HTTP API contracts, routers, schemas, or API smoke behavior.
 area/auth
Authentication, sessions, access bootstrap, or login behavior.
 area/db
Database sessions, models, transactions, or persistence primitives.
 area/devex
Local developer workflow, scripts, tests, tooling, or release helpers.
 area/docs
Durable documentation and project guidance.
 area/governance
Governance policy, audit, privacy, retention, or compliance behavior.
 area/marketing
Public website, product messaging, publication copy, or legal page content.
 area/migrations
Alembic migrations, schema bootstrap, or persistence evolution.
 area/module-system
Module discovery, manifests, capabilities, routing, or optional integrations.
 area/rbac
Permissions, roles, delegation, or authorization policy.
 area/release
Versioning, release locks, tags, packaging, or dependency pins.
 area/security
Security posture, static analysis, supply-chain hardening, or vulnerability remediation.
 area/tenancy
Tenant boundaries, provisioning, or tenant-scoped data behavior.
 area/webui
Shared WebUI shell, frontend components, routing, or frontend tests.
 audit/complexity
Complexity finding from Radon, Xenon, or equivalent maintainability scans.
 audit/duplication
Duplicated-code finding from jscpd or equivalent similarity scans.
 audit/false-positive
Audit finding reviewed as a narrow false positive or acceptable risk.
 audit/needs-design
Audit finding that needs an architectural or product decision before implementation.
 audit/quick-fix
Audit finding that appears narrow and directly fixable.
 audit/structural
Audit finding that needs design, refactoring, or behavior review.
 codex/needs-human
Needs an explicit human decision before Codex should implement.
 codex/ready
Suitable for Codex to pick up with the existing issue context.
 module/access
GovOPlaN access, identity, authentication, RBAC, and administration behavior.
 module/addresses
GovOPlaN Addresses module behavior or integration.
 module/admin
GovOPlaN Admin module behavior or integration.
 module/appointments
GovOPlaN Appointments module behavior or integration.
 module/approvals
GovOPlaN Approvals module behavior or integration.
 module/audit
GovOPlaN Audit module behavior or integration.
 module/calendar
GovOPlaN Calendar module behavior or integration.
 module/campaign
GovOPlaN campaign module behavior or integration.
 module/cases
GovOPlaN Cases module behavior or integration.
 module/committee
GovOPlaN Committee module behavior or integration.
 module/connectors
GovOPlaN Connectors module behavior or integration.
 module/core
GovOPlaN core runner, shared primitives, shell, or extension points.
 module/dashboard
GovOPlaN Dashboard module behavior or integration.
 module/dataflow
GovOPlaN Dataflow module behavior or integration.
 module/datasources
GovOPlaN governed datasource contracts, catalogs, and integrations.
 module/decisions
GovOPlaN formal Decisions module behavior or integration.
 module/dist-lists
GovOPlaN Distribution Lists module behavior or integration.
 module/dms
GovOPlaN Dms module behavior or integration.
 module/docs
GovOPlaN Docs module behavior or integration.
 module/encryption
GovOPlaN Encryption key custody, cryptographic policy, and E2EE integration.
 module/erp
GovOPlaN Erp module behavior or integration.
 module/evaluation
GovOPlaN Evaluation module behavior or integration.
 module/files
GovOPlaN files module behavior or integration.
 module/fit-connect
GovOPlaN Fit Connect module behavior or integration.
 module/forms
GovOPlaN Forms module behavior or integration.
 module/forms-runtime
GovOPlaN Forms Runtime module behavior or integration.
 module/helpdesk
GovOPlaN Helpdesk module behavior or integration.
 module/identity
GovOPlaN Identity module behavior or integration.
 module/identity-trust
GovOPlaN Identity Trust module behavior or integration.
 module/idm
GovOPlaN Idm module behavior or integration.
 module/ledger
GovOPlaN Ledger module behavior or integration.
 module/mail
GovOPlaN mail module behavior or integration.
 module/mandates
GovOPlaN Mandates, jurisdiction, responsibility, and authority behavior or integration.
 module/notifications
GovOPlaN Notifications module behavior or integration.
 module/ops
GovOPlaN Ops module behavior or integration.
 module/organizations
GovOPlaN Organizations module behavior or integration.
 module/parties
GovOPlaN procedure Parties, representation, and delivery-authority behavior or integration.
 module/payments
GovOPlaN Payments module behavior or integration.
 module/permits
GovOPlaN Permits module behavior or integration.
 module/policy
GovOPlaN Policy module behavior or integration.
 module/poll
GovOPlaN Poll module behavior or integration.
 module/portal
GovOPlaN Portal module behavior or integration.
 module/postbox
GovOPlaN Postbox module behavior or integration.
 module/projects
GovOPlaN Projects module behavior or integration.
 module/quick-access
GovOPlaN configurable task-local Quick Access behavior and integrations.
 module/records
GovOPlaN Records and eAkte lifecycle behavior or integration.
 module/reporting
GovOPlaN Reporting module behavior or integration.
 module/risk-compliance
GovOPlaN Risk Compliance module behavior or integration.
 module/scheduling
GovOPlaN Scheduling module behavior or integration.
 module/search
GovOPlaN Search module behavior or integration.
 module/services
GovOPlaN versioned institutional Services behavior or integration.
 module/tasks
GovOPlaN Tasks module behavior or integration.
 module/templates
GovOPlaN Templates module behavior or integration.
 module/tenancy
GovOPlaN Tenancy module behavior or integration.
 module/tickets
GovOPlaN Tickets module behavior or integration.
 module/views
GovOPlaN governed task views, interface projections, and workflow view integration.
 module/voting
GovOPlaN Voting module behavior or integration.
 module/wiki
GovOPlaN Wiki module behavior or integration.
 module/workflow
GovOPlaN Workflow module behavior or integration.
 module/workflow-engine
GovOPlaN Workflow Engine runtime, persistence, or integration.
 module/xoev
GovOPlaN Xoev module behavior or integration.
 module/xrechnung
GovOPlaN Xrechnung module behavior or integration.
 module/xta-osci
GovOPlaN Xta Osci module behavior or integration.
priority
p0
Immediate stop-the-line priority.
priority
p1
High priority for the next focused work window.
priority
p2
Normal planned priority.
priority
p3
Low priority or opportunistic cleanup.
source/backlog-import
Imported from markdown backlog, roadmap, plan, or TODO files.
 source/security-audit
Created from a structured security or code-quality audit report.
 source/todo-scan
Imported from inline TODO/FIXME/HACK markers by the Gitea TODO importer.
status
blocked
Cannot progress without a decision, dependency, credential, or external change.
status
in-progress
Currently being worked.
status
needs-info
Needs clarifying input before implementation can proceed safely.
status
ready
Ready for implementation.
status
triage
Needs review, ownership, priority, or acceptance criteria.
type
bug
A reproducible defect, regression, or incorrect behavior.
type
debt
Cleanup, refactoring, risk reduction, or deferred engineering work.
type
docs
Documentation, process, or developer workflow work.
type
feature
New user-visible behavior or platform capability.
type
task
Implementation, maintenance, migration, or operational work.
type
user-story
End-to-end user journey or real-world process story used to steer product slices.
No labels area/api area/db area/webui codex/ready module/datasources
priority
p1
status
in-progress
type
feature
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
govoplan-package-publisher zemion (Albrecht Degering)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-datasources#1