GovOPlaN Dataflow defines and runs governed tabular transformation pipelines. Power users can work with the same pipeline as a graphical node graph or as a constrained SQL query. Every saved definition change produces an immutable revision, and every preview records diagnostics and reproducibility metadata without storing the previewed row contents.

Boundary

Dataflow: pipeline graphs, revisions, validation, constrained SQL, previews/runs, diagnostics, and lineage.
Datasources: governed source identity, staging, materializations, frozen states, and bounded source access.
Connectors: external acquisition, credentials, discovery, and provider health.
Reporting: governed datasets, analytical views, dashboards, and exports.
Workflow: orchestration, resumability, approvals, and module handoffs.
Risk Compliance: sanctions matching policy, review, dispositions, and legal evidence.

Node Library

The canonical backend catalogue is exposed to the WebUI and groups executable nodes by purpose:

Group	Nodes
Load	Inline data, datasource
Combine	Append rows, join tables
Filter	Filter rows, remove duplicates
Transform	Select columns, derive column, aggregate, sort, limit
Output	Preview output

Join nodes have explicit left and right ports. Append nodes accept two or more inputs. Derived columns use a constrained operation catalogue rather than arbitrary code. The graph editor validates manual, proximity-created, and reconnected edges against the same port, multiplicity, and cycle rules. SQL round-trips retain node and edge identity, labels, and coordinates while the graph topology remains compatible. Structural changes use a compact layered layout with bounded branch and edge spacing.

Governed inputs are resolved through the versioned Core capability datasources.catalogue; Dataflow imports neither Datasources nor Connectors and stores only opaque datasource references plus expected fingerprints. A source node can request the current, live, or latest frozen state. Fingerprint drift fails visibly instead of silently changing a run.

SQL And Preview Safety

The SQL workbench parses one SELECT or column-aligned UNION BY NAME statement into the canonical graph. The dialect supports projection, aliases, filters, grouping, aggregate functions, sorting, limits, DISTINCT, append, and one two-source equi-join. It rejects DDL, DML, arbitrary subqueries, arbitrary functions, file access, and unchecked pass-through execution.

Preview reads at most 250 rows per source and enforces time, intermediate-row, result-byte, graph-node, and response-row bounds. Saved previews record the pipeline revision, executor version, source fingerprints, node diagnostics, and output summary, but not source or result rows. A preview may return one explicitly requested intermediate node state under the same response bound; that row payload remains ephemeral and is not added to run evidence. The WebUI targets the selected graph node and supports remembered automatic or manual-refresh preview modes.

Saved revisions can also be started through the versioned dataflow.runLifecycle capability or the Run dialog. The first runner is synchronous and bounded. It records an idempotent terminal run, pins the pipeline revision, and can atomically publish a complete result through datasources.publication. Publication is rejected when a source or result was truncated; larger asynchronous and artifact-backed execution belongs to a subsequent runner slice.

Governed Definitions And Automation

Pipelines can be complete flows or reusable templates at system, tenant, group, or user scope. Templates never run directly. A permitted consumer derives a new definition that pins the source revision and content hash and records the effective Policy decision and ancestor limits. Inherited definitions remain read-only; lower scopes may narrow, but not broaden, execution, reuse, inheritance, or automation permissions.

Complete active flows support explicit user/API starts, administrative backfills, one-time schedules, interval schedules, and exact-match platform events. Trigger deliveries are durable and idempotent. They pin the pipeline revision and a least-privilege scope grant, then ask Access to rebuild the owner's current principal before every run. Revoked memberships or reduced permissions block the delivery before source access or output publication. Confidential and restricted events are not accepted through the direct ingress; those require Core's transactional event bridge.

Development

/mnt/DATA/git/govoplan/.venv/bin/python -m pip install -e .
/mnt/DATA/git/govoplan/.venv/bin/python -m unittest discover -s tests

WebUI:

cd webui
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:$PATH npm run typecheck
npm run test:structure

The implementation epic is govoplan-dataflow#1.