GovOPlaN Dataflow defines and runs governed tabular transformation pipelines. Power users can work with the same pipeline as a graphical node graph or as a constrained SQL query. Every saved definition change produces an immutable revision, and every preview records diagnostics and reproducibility metadata without storing the previewed row contents.

Boundary

Dataflow: pipeline graphs, revisions, validation, constrained SQL, previews/runs, diagnostics, and lineage.
Datasources: governed source identity, staging, materializations, frozen states, and bounded source access.
Connectors: external acquisition, credentials, discovery, and provider health.
Reporting: governed datasets, analytical views, dashboards, and exports.
Workflow: orchestration, resumability, approvals, and module handoffs.
Risk Compliance: sanctions matching policy, review, dispositions, and legal evidence.

Node Library

The canonical backend catalogue is exposed to the WebUI and groups executable nodes by purpose:

Group	Nodes
Load	Inline data, datasource
Combine	Append rows, join tables
Filter	Filter rows, remove duplicates
Transform	Select columns, derive column, aggregate, sort, limit
Output	Preview output

Join nodes have explicit left and right ports. Append nodes accept two or more inputs. Derived columns use a constrained operation catalogue rather than arbitrary code.

Governed inputs are resolved through the versioned Core capability datasources.catalogue; Dataflow imports neither Datasources nor Connectors and stores only opaque datasource references plus expected fingerprints. A source node can request the current, live, or latest frozen state. Fingerprint drift fails visibly instead of silently changing a run.

SQL And Preview Safety

The SQL workbench parses one SELECT statement into the canonical graph. The dialect supports projection, aliases, filters, grouping, aggregate functions, sorting, limits, DISTINCT, and one two-source equi-join. It rejects DDL, DML, subqueries, arbitrary functions, file access, and unchecked pass-through execution.

Preview reads at most 250 rows per source and enforces time, intermediate-row, result-byte, graph-node, and response-row bounds. Saved previews record the pipeline revision, executor version, source fingerprints, node diagnostics, and output summary, but not source or result rows.

Saved revisions can also be started through the versioned dataflow.runLifecycle capability or the Run dialog. The first runner is synchronous and bounded. It records an idempotent terminal run, pins the pipeline revision, and can atomically publish a complete result through datasources.publication. Publication is rejected when a source or result was truncated; larger asynchronous and artifact-backed execution belongs to a subsequent runner slice.

Development

/mnt/DATA/git/govoplan/.venv/bin/python -m pip install -e .
/mnt/DATA/git/govoplan/.venv/bin/python -m unittest discover -s tests

WebUI:

cd webui
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:$PATH npm run typecheck
npm run test:structure

The implementation epic is govoplan-dataflow#1.