GovOPlaN/govoplan-dataflow
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

[Epic] Deliver a governed Dataflow manager with graphical and SQL editing #1

New Issue
Open
opened 2026-07-28 00:58:08 +02:00 by zemion · 8 comments
zemion commented 2026-07-28 00:58:08 +02:00
Owner
Copy Link

Outcome

Deliver a production-shaped Dataflow module that lets power users define a transformation once, inspect it either as a directed graph or constrained SQL, preview its result, and preserve the exact definition and diagnostics used for every run.

This is the enabling capability for the monthly-data user story, governed BI paths, and sanctions-source normalization. It does not own external connections, legal screening decisions, report presentation, or workflow orchestration.

Boundary

  • Dataflow owns canonical pipeline graphs, versions, validation, compilation, preview/run state, diagnostics, and lineage.
  • Connectors own connection profiles, source acquisition, source health, and schema discovery.
  • Reporting owns published analytical views, dashboards, and exports.
  • Workflow owns durable orchestration, approvals, resumability, and cross-module handoffs.
  • Risk Compliance owns sanctions semantics, match review, exceptions, and policy gates.

First Release Gate

  • Persist tenant-isolated pipeline definitions and immutable revisions.
  • Edit the same supported pipeline through a graphical node canvas and constrained SQL.
  • Validate cycles, ports, schemas, unsupported SQL, and unsafe operations before execution.
  • Run a bounded preview over an inline or connector-provided tabular source.
  • Show row/column output plus per-node diagnostics.
  • Record definition revision, input fingerprint, executor version, diagnostics, and output summary.
  • Enforce read/write/run/admin permissions and audit state-changing actions.
  • Automated backend and WebUI tests cover the first end-to-end path.

Product References

## Outcome Deliver a production-shaped Dataflow module that lets power users define a transformation once, inspect it either as a directed graph or constrained SQL, preview its result, and preserve the exact definition and diagnostics used for every run. This is the enabling capability for the monthly-data user story, governed BI paths, and sanctions-source normalization. It does not own external connections, legal screening decisions, report presentation, or workflow orchestration. ## Boundary - Dataflow owns canonical pipeline graphs, versions, validation, compilation, preview/run state, diagnostics, and lineage. - Connectors own connection profiles, source acquisition, source health, and schema discovery. - Reporting owns published analytical views, dashboards, and exports. - Workflow owns durable orchestration, approvals, resumability, and cross-module handoffs. - Risk Compliance owns sanctions semantics, match review, exceptions, and policy gates. ## First Release Gate - [x] Persist tenant-isolated pipeline definitions and immutable revisions. - [x] Edit the same supported pipeline through a graphical node canvas and constrained SQL. - [x] Validate cycles, ports, schemas, unsupported SQL, and unsafe operations before execution. - [x] Run a bounded preview over an inline or connector-provided tabular source. - [x] Show row/column output plus per-node diagnostics. - [x] Record definition revision, input fingerprint, executor version, diagnostics, and output summary. - [x] Enforce read/write/run/admin permissions and audit state-changing actions. - [x] Automated backend and WebUI tests cover the first end-to-end path. ## Product References - [Monthly guided data workflow](https://git.add-ideas.de/GovOPlaN/govoplan/issues/8) - [Sanctions screening](https://git.add-ideas.de/GovOPlaN/govoplan/issues/12) - [Connected platform source-to-indicator stage](https://git.add-ideas.de/GovOPlaN/govoplan/issues/14)
zemion commented 2026-07-28 01:32:40 +02:00
Author
Owner
Copy Link

Codex State: progress

Summary

  • Initialized and pushed the standalone Dataflow module on main at df468a2.
  • Implemented tenant-isolated versioned pipelines, constrained SQL/graph conversion, bounded inline previews, run evidence, permissions, audit events, and a full-height dual-mode WebUI.

Changed Files

  • pyproject.toml
  • src/govoplan_dataflow/backend
  • webui/src
  • tests

Verification

  • 15 Dataflow backend tests pass
  • Ruff passes
  • standalone TypeScript typecheck and structure test pass
  • core Dataflow-only and full-product production builds pass
  • consolidated focused check passes across 44 manifests and all module permutations

Next / Blocked

  • Connectors #9 and Dataflow #6 provide the next vertical slice: provider-neutral source discovery and bounded connector reads.
  • Keep SQL #3, preview #4, and WebUI #5 open for their recorded hardening criteria.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Initialized and pushed the standalone Dataflow module on main at df468a2. - Implemented tenant-isolated versioned pipelines, constrained SQL/graph conversion, bounded inline previews, run evidence, permissions, audit events, and a full-height dual-mode WebUI. ### Changed Files - `pyproject.toml` - `src/govoplan_dataflow/backend` - `webui/src` - `tests` ### Verification - `15 Dataflow backend tests pass` - `Ruff passes` - `standalone TypeScript typecheck and structure test pass` - `core Dataflow-only and full-product production builds pass` - `consolidated focused check passes across 44 manifests and all module permutations` ### Next / Blocked - Connectors #9 and Dataflow #6 provide the next vertical slice: provider-neutral source discovery and bounded connector reads. - Keep SQL #3, preview #4, and WebUI #5 open for their recorded hardening criteria. Suggested status label: `status/in-progress`
zemion commented 2026-07-28 01:34:33 +02:00
Author
Owner
Copy Link

Codex State: progress

Summary

  • Published the module at df468a2, core registration at c7183fe, and meta repository/development composition at 97dfd33.
  • The Dataflow README is mirrored to its Gitea wiki; core#189 is closed in favor of this dedicated epic.

Verification

  • All three published repositories are clean relative to origin; unrelated paused credential work remains uncommitted in core.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Published the module at df468a2, core registration at c7183fe, and meta repository/development composition at 97dfd33. - The Dataflow README is mirrored to its Gitea wiki; core#189 is closed in favor of this dedicated epic. ### Verification - `All three published repositories are clean relative to origin; unrelated paused credential work remains uncommitted in core.` Suggested status label: `status/in-progress`
zemion commented 2026-07-28 11:18:40 +02:00
Author
Owner
Copy Link

Codex State: progress

Summary

  • Published the provider-neutral core contract (govoplan-core@d36bb94), executable snapshot provider (govoplan-connectors@ba5ccea), and expanded editor/runtime (govoplan-dataflow@dee8380).
  • The backend-owned library now exposes 12 reusable nodes grouped as Load, Combine, Filter, Transform, and Output.
  • JSON/CSV snapshot import, schema/fingerprint discovery, bounded connector reads, static schema validation, SQL joins/DISTINCT, node diagnostics, and response/run lineage are integrated.
  • The meta startup, release, and focused CI paths include Connectors and Dataflow (govoplan@603e07c).

Verification

  • python -m unittest discover -s govoplan-connectors/tests (7 passed)
  • python -m unittest discover -s govoplan-dataflow/tests (24 passed)
  • tools/checks/check-contracts.sh (45 contracts passed)
  • tools/checks/check-focused.sh (core, module, API, and WebUI checks passed)
  • Composed JSON/CSV snapshot and connector-backed preview probe passed.

Next / Blocked

  • Complete #3 source-location diagnostics, #5 accessibility/interaction verification, and #6 pushdown/provider follow-up before closing the epic.
## Codex State: progress ### Summary - Published the provider-neutral core contract (`govoplan-core@d36bb94`), executable snapshot provider (`govoplan-connectors@ba5ccea`), and expanded editor/runtime (`govoplan-dataflow@dee8380`). - The backend-owned library now exposes 12 reusable nodes grouped as Load, Combine, Filter, Transform, and Output. - JSON/CSV snapshot import, schema/fingerprint discovery, bounded connector reads, static schema validation, SQL joins/DISTINCT, node diagnostics, and response/run lineage are integrated. - The meta startup, release, and focused CI paths include Connectors and Dataflow (`govoplan@603e07c`). ### Verification - `python -m unittest discover -s govoplan-connectors/tests` (7 passed) - `python -m unittest discover -s govoplan-dataflow/tests` (24 passed) - `tools/checks/check-contracts.sh` (45 contracts passed) - `tools/checks/check-focused.sh` (core, module, API, and WebUI checks passed) - Composed JSON/CSV snapshot and connector-backed preview probe passed. ### Next / Blocked - Complete #3 source-location diagnostics, #5 accessibility/interaction verification, and #6 pushdown/provider follow-up before closing the epic.
zemion commented 2026-07-28 12:48:20 +02:00
Author
Owner
Copy Link

Codex State: progress

Dataflow now uses Core's shared definition-graph primitives and consumes the
governed Datasources catalogue/lifecycle contract rather than Connectors
directly (6ca3058). The source library supports current, direct-live, and
latest-frozen reads with expected-fingerprint drift checks.

Verification includes 24 backend tests, WebUI TypeScript/build checks, static
manifest/contract gates, and a durable Connector -> Datasources -> Dataflow
composition check. Browser-level editor E2E coverage and production run/output
publication remain open.

## Codex State: progress Dataflow now uses Core's shared definition-graph primitives and consumes the governed Datasources catalogue/lifecycle contract rather than Connectors directly (`6ca3058`). The source library supports current, direct-live, and latest-frozen reads with expected-fingerprint drift checks. Verification includes 24 backend tests, WebUI TypeScript/build checks, static manifest/contract gates, and a durable Connector -> Datasources -> Dataflow composition check. Browser-level editor E2E coverage and production run/output publication remain open.
zemion commented 2026-07-28 13:50:40 +02:00
Author
Owner
Copy Link

Codex State: progress

Summary

  • Extended the first release path with persisted pinned runs, complete-result publication, immutable output references, and a Run/Publish WebUI in 6305ef9.

Verification

  • 27 Dataflow tests passed; cross-module composition and all WebUI module permutations passed.

Next / Blocked

  • The release gate remains open until a WebUI behavior test covers the end-to-end author/save/run/publish path; current WebUI verification is typecheck/build coverage.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Extended the first release path with persisted pinned runs, complete-result publication, immutable output references, and a Run/Publish WebUI in 6305ef9. ### Verification - `27 Dataflow tests passed; cross-module composition and all WebUI module permutations passed.` ### Next / Blocked - The release gate remains open until a WebUI behavior test covers the end-to-end author/save/run/publish path; current WebUI verification is typecheck/build coverage. Suggested status label: `status/in-progress`
zemion commented 2026-07-28 16:32:54 +02:00
Author
Owner
Copy Link

Codex State: progress

Summary

  • Added bounded, ephemeral preview results for any explicitly selected pipeline node, including upstream state retained when a later node fails.
  • Added inspector and results-stage controls for intermediate-state inspection while preserving terminal output as run evidence.
  • Added validated edge reconnection, disconnect-on-empty-drop, keyboard deletion, and dotted proximity connection candidates.

Changed Files

  • src/govoplan_dataflow/backend/{schemas,executor,service,router}.py
  • webui/src/features/dataflow/{DataflowPage,DataflowCanvas,NodeInspector,graphInteractions}.tsx

Verification

  • 36 Dataflow backend tests passed
  • Dataflow WebUI typecheck and structure checks passed
  • GovOPlaN focused suite, all module permutations, and full-product build passed

Next / Blocked

  • Commit 521829a is pushed to main; perform normal browser UX validation of the proximity threshold and reconnect handles.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Added bounded, ephemeral preview results for any explicitly selected pipeline node, including upstream state retained when a later node fails. - Added inspector and results-stage controls for intermediate-state inspection while preserving terminal output as run evidence. - Added validated edge reconnection, disconnect-on-empty-drop, keyboard deletion, and dotted proximity connection candidates. ### Changed Files - `src/govoplan_dataflow/backend/{schemas,executor,service,router}.py` - `webui/src/features/dataflow/{DataflowPage,DataflowCanvas,NodeInspector,graphInteractions}.tsx` ### Verification - `36 Dataflow backend tests passed` - `Dataflow WebUI typecheck and structure checks passed` - `GovOPlaN focused suite, all module permutations, and full-product build passed` ### Next / Blocked - Commit 521829a is pushed to main; perform normal browser UX validation of the proximity threshold and reconnect handles. Suggested status label: `status/in-progress`
zemion commented 2026-07-28 17:54:59 +02:00
Author
Owner
Copy Link

Codex State: progress

Summary

  • Fixed oversized composite edge IDs: new graph edges use opaque UUID IDs, while legacy IDs up to the previous worst-case size remain accepted.
  • Added constrained UNION BY NAME graph-to-SQL generation and SQL-to-graph round-tripping for direct append sources and downstream transforms.
  • Intermediate previews now follow the selected node; a remembered Manual/Auto toggle controls explicit refresh versus debounced generation, and the stage dropdown was removed.
  • Dataflow page notices now use Core's shared floating DismissibleAlert stack.

Changed Files

  • src/govoplan_dataflow/backend/sql_compiler.py
  • webui/src/features/dataflow/DataflowPage.tsx
  • webui/src/features/dataflow/DataflowCanvas.tsx

Verification

  • 37 Dataflow backend tests passed; Ruff passed.
  • Dataflow WebUI typecheck and structure tests passed.
  • Full tools/checks/check-focused.sh integration and WebUI module permutation battery passed.

Next / Blocked

  • Continue the open epic with richer branch-level SQL shapes and artifact-backed execution as separately scoped slices.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Fixed oversized composite edge IDs: new graph edges use opaque UUID IDs, while legacy IDs up to the previous worst-case size remain accepted. - Added constrained UNION BY NAME graph-to-SQL generation and SQL-to-graph round-tripping for direct append sources and downstream transforms. - Intermediate previews now follow the selected node; a remembered Manual/Auto toggle controls explicit refresh versus debounced generation, and the stage dropdown was removed. - Dataflow page notices now use Core's shared floating DismissibleAlert stack. ### Changed Files - `src/govoplan_dataflow/backend/sql_compiler.py` - `webui/src/features/dataflow/DataflowPage.tsx` - `webui/src/features/dataflow/DataflowCanvas.tsx` ### Verification - `37 Dataflow backend tests passed; Ruff passed.` - `Dataflow WebUI typecheck and structure tests passed.` - `Full tools/checks/check-focused.sh integration and WebUI module permutation battery passed.` ### Next / Blocked - Continue the open epic with richer branch-level SQL shapes and artifact-backed execution as separately scoped slices. Suggested status label: `status/in-progress`
zemion commented 2026-07-28 18:37:17 +02:00
Author
Owner
Copy Link

Codex State: progress

Summary

  • Preserved node IDs, labels, edge IDs, ordering, selection, and coordinates when SQL recompilation keeps a structurally compatible graph, including non-topological storage order.
  • Changed SQL topology now receives a compact layer-based layout with 240px node steps, leaving a 60px edge corridor for 180px nodes, and 120px branch spacing.
  • Removed the Dataflow results-toolbar flex-gap override from the shared SegmentedControl.
  • Fixed transparent success/info DismissibleAlert tones centrally in govoplan-core commit 3f58702.

Changed Files

  • src/govoplan_dataflow/backend/graph.py
  • src/govoplan_dataflow/backend/sql_compiler.py
  • webui/src/features/dataflow/DataflowPage.tsx
  • webui/src/styles/dataflow.css

Verification

  • 39 Dataflow backend tests and Ruff passed.
  • Dataflow WebUI typecheck and structure tests passed.
  • Core full-product build and full tools/checks/check-focused.sh battery passed.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Preserved node IDs, labels, edge IDs, ordering, selection, and coordinates when SQL recompilation keeps a structurally compatible graph, including non-topological storage order. - Changed SQL topology now receives a compact layer-based layout with 240px node steps, leaving a 60px edge corridor for 180px nodes, and 120px branch spacing. - Removed the Dataflow results-toolbar flex-gap override from the shared SegmentedControl. - Fixed transparent success/info DismissibleAlert tones centrally in govoplan-core commit 3f58702. ### Changed Files - `src/govoplan_dataflow/backend/graph.py` - `src/govoplan_dataflow/backend/sql_compiler.py` - `webui/src/features/dataflow/DataflowPage.tsx` - `webui/src/styles/dataflow.css` ### Verification - `39 Dataflow backend tests and Ruff passed.` - `Dataflow WebUI typecheck and structure tests passed.` - `Core full-product build and full tools/checks/check-focused.sh battery passed.` Suggested status label: `status/in-progress`
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/api
HTTP API contracts, routers, schemas, or API smoke behavior.
area/auth
Authentication, sessions, access bootstrap, or login behavior.
area/db
Database sessions, models, transactions, or persistence primitives.
area/devex
Local developer workflow, scripts, tests, tooling, or release helpers.
area/docs
Durable documentation and project guidance.
area/governance
Governance policy, audit, privacy, retention, or compliance behavior.
area/marketing
Public website, product messaging, publication copy, or legal page content.
area/migrations
Alembic migrations, schema bootstrap, or persistence evolution.
area/module-system
Module discovery, manifests, capabilities, routing, or optional integrations.
area/rbac
Permissions, roles, delegation, or authorization policy.
area/release
Versioning, release locks, tags, packaging, or dependency pins.
area/security
Security posture, static analysis, supply-chain hardening, or vulnerability remediation.
area/tenancy
Tenant boundaries, provisioning, or tenant-scoped data behavior.
area/webui
Shared WebUI shell, frontend components, routing, or frontend tests.
audit/complexity
Complexity finding from Radon, Xenon, or equivalent maintainability scans.
audit/duplication
Duplicated-code finding from jscpd or equivalent similarity scans.
audit/false-positive
Audit finding reviewed as a narrow false positive or acceptable risk.
audit/needs-design
Audit finding that needs an architectural or product decision before implementation.
audit/quick-fix
Audit finding that appears narrow and directly fixable.
audit/structural
Audit finding that needs design, refactoring, or behavior review.
codex/needs-human
Needs an explicit human decision before Codex should implement.
codex/ready
Suitable for Codex to pick up with the existing issue context.
module/access
GovOPlaN access, identity, authentication, RBAC, and administration behavior.
module/addresses
GovOPlaN Addresses module behavior or integration.
module/admin
GovOPlaN Admin module behavior or integration.
module/appointments
GovOPlaN Appointments module behavior or integration.
module/audit
GovOPlaN Audit module behavior or integration.
module/calendar
GovOPlaN Calendar module behavior or integration.
module/campaign
GovOPlaN campaign module behavior or integration.
module/cases
GovOPlaN Cases module behavior or integration.
module/connectors
GovOPlaN Connectors module behavior or integration.
module/core
GovOPlaN core runner, shared primitives, shell, or extension points.
module/dataflow
GovOPlaN Dataflow module behavior or integration.
module/datasources
GovOPlaN governed datasource contracts, catalogs, and integrations.
module/dist-lists
GovOPlaN Distribution Lists module behavior or integration.
module/dms
GovOPlaN Dms module behavior or integration.
module/erp
GovOPlaN Erp module behavior or integration.
module/evaluation
GovOPlaN Evaluation module behavior or integration.
module/files
GovOPlaN files module behavior or integration.
module/fit-connect
GovOPlaN Fit Connect module behavior or integration.
module/forms
GovOPlaN Forms module behavior or integration.
module/identity
GovOPlaN Identity module behavior or integration.
module/identity-trust
GovOPlaN Identity Trust module behavior or integration.
module/idm
GovOPlaN Idm module behavior or integration.
module/ledger
GovOPlaN Ledger module behavior or integration.
module/mail
GovOPlaN mail module behavior or integration.
module/notifications
GovOPlaN Notifications module behavior or integration.
module/ops
GovOPlaN Ops module behavior or integration.
module/organizations
GovOPlaN Organizations module behavior or integration.
module/payments
GovOPlaN Payments module behavior or integration.
module/permits
GovOPlaN Permits module behavior or integration.
module/policy
GovOPlaN Policy module behavior or integration.
module/poll
GovOPlaN Poll module behavior or integration.
module/portal
GovOPlaN Portal module behavior or integration.
module/postbox
GovOPlaN Postbox module behavior or integration.
module/reporting
GovOPlaN Reporting module behavior or integration.
module/risk-compliance
GovOPlaN Risk Compliance module behavior or integration.
module/scheduling
GovOPlaN Scheduling module behavior or integration.
module/search
GovOPlaN Search module behavior or integration.
module/tasks
GovOPlaN Tasks module behavior or integration.
module/templates
GovOPlaN Templates module behavior or integration.
module/tenancy
GovOPlaN Tenancy module behavior or integration.
module/views
GovOPlaN governed task views, interface projections, and workflow view integration.
module/workflow
GovOPlaN Workflow module behavior or integration.
module/xoev
GovOPlaN Xoev module behavior or integration.
module/xrechnung
GovOPlaN Xrechnung module behavior or integration.
module/xta-osci
GovOPlaN Xta Osci module behavior or integration.
priority
p0
Immediate stop-the-line priority.
priority
p1
High priority for the next focused work window.
priority
p2
Normal planned priority.
priority
p3
Low priority or opportunistic cleanup.
source/backlog-import
Imported from markdown backlog, roadmap, plan, or TODO files.
source/security-audit
Created from a structured security or code-quality audit report.
source/todo-scan
Imported from inline TODO/FIXME/HACK markers by the Gitea TODO importer.
status
blocked
Cannot progress without a decision, dependency, credential, or external change.
status
in-progress
Currently being worked.
status
needs-info
Needs clarifying input before implementation can proceed safely.
status
ready
Ready for implementation.
status
triage
Needs review, ownership, priority, or acceptance criteria.
type
bug
A reproducible defect, regression, or incorrect behavior.
type
debt
Cleanup, refactoring, risk reduction, or deferred engineering work.
type
docs
Documentation, process, or developer workflow work.
type
feature
New user-visible behavior or platform capability.
type
task
Implementation, maintenance, migration, or operational work.
type
user-story
End-to-end user journey or real-world process story used to steer product slices.
No Label
area/api
area/db
area/webui
codex/ready
module/dataflow
priority
p1
status
in-progress
type
feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
zemion
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-dataflow#1
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?