GovOPlaN Dataflow defines and runs governed tabular transformation pipelines. Power users can work with the same pipeline as a graphical node graph or as a constrained SQL query. Every saved definition change produces an immutable revision, and every preview records diagnostics and reproducibility metadata without storing the previewed row contents.

Boundary

Dataflow: pipeline graphs, revisions, validation, constrained SQL, previews/runs, diagnostics, and lineage.
Datasources: governed source identity, staging, materializations, frozen states, and bounded source access.
Connectors: external acquisition, credentials, discovery, and provider health.
Reporting: governed datasets, analytical views, dashboards, and exports.
Workflow: orchestration, resumability, approvals, and module handoffs.
Risk Compliance: sanctions matching policy, review, dispositions, and legal evidence.

Node Library

The canonical backend catalogue is exposed to the WebUI and groups executable nodes by purpose:

Group	Nodes
Load	Inline data, datasource
Combine	Append rows, join tables
Filter	Column filter, typed-expression filter, remove duplicates
Transform	Select, derive, typed expression, conversion, replacement, aggregate, sort, limit, reusable subflow
Quality	Quality rules, keyed reconciliation
Output	Preview output

Join nodes have explicit left and right ports. Append nodes accept two or more inputs. Derived columns use a constrained operation catalogue rather than arbitrary code. The graph editor validates manual, proximity-created, and reconnected edges against the same port, multiplicity, and cycle rules. SQL round-trips retain node and edge identity, labels, and coordinates while the graph topology remains compatible. Structural changes use a compact layered layout with bounded branch and edge spacing.

Governed inputs are resolved through the versioned Core capability datasources.catalogue; Dataflow imports neither Datasources nor Connectors and stores only opaque datasource references plus expected fingerprints. A source node can request the current, live, or latest frozen state. Fingerprint drift fails visibly instead of silently changing a run.

SQL And Preview Safety

The SQL workbench parses one SELECT or column-aligned UNION BY NAME statement into the canonical graph. The dialect supports projection, aliases, filters, grouping, aggregate functions, sorting, limits, DISTINCT, append, and one two-source equi-join. It rejects DDL, DML, arbitrary subqueries, arbitrary functions, file access, and unchecked pass-through execution. The typed-expression library is a separate allowlisted AST evaluator shared by expression filters and calculated columns. It supports literals, columns, arithmetic, comparisons, boolean logic, CASE, safe casts, and a bounded string/numeric function catalogue without Python evaluation or effectful SQL.

Node definitions, validators, schema propagators, preview executors, and SQL renderers are registered independently in the operator registry. Adding a node no longer requires another branch in the preview or graph-to-SQL dispatch loop. Operators that cannot be represented by constrained SQL declare that explicitly.

Every executable graph also has a versioned typed IR. It preserves graph identity and layout while giving ports, schemas, expressions, parameters, lineage, diagnostics, semantic hashes, and physical results stable contracts. The backend boundary exchanges bounded typed columnar batches that can be serialized as Arrow IPC. The deterministic Python executor remains the default and fallback.

Install .[analytics] to enable the analytical backend. It executes only SQL generated from validated graphs in a separate short-lived DuckDB process. Inputs and outputs cross the process boundary as Arrow IPC; external access, extension installation/loading, persistent secrets, configuration changes, temporary spill files, and multiple DuckDB threads are disabled. Wall-clock, row, byte, memory, file-descriptor, and backend-concurrency limits are applied. Callers can explicitly request reference, duckdb, or auto for a preview.

Preview reads at most 250 rows per source and enforces time, intermediate-row, result-byte, graph-node, and response-row bounds. Saved previews record the pipeline revision, executor version, source fingerprints, node diagnostics, and output summary, but not source or result rows. A preview may return one explicitly requested intermediate node state under the same response bound; that row payload remains ephemeral and is not added to run evidence. The WebUI targets the selected graph node and supports remembered automatic or manual-refresh preview modes.

Saved revisions can also be started through the versioned dataflow.runLifecycle capability or the Run dialog. The first runner is synchronous and bounded. It records an idempotent terminal run, pins the pipeline revision, and can atomically publish a complete result through datasources.publication. Publication is rejected when a source or result was truncated; larger asynchronous and artifact-backed execution belongs to a subsequent runner slice.

Governed Definitions And Automation

Pipelines can be complete flows or reusable templates at system, tenant, group, or user scope. Templates never run directly. A permitted consumer derives a new definition that pins the source revision and content hash and records the effective Policy decision and ancestor limits. Inherited definitions remain read-only; lower scopes may narrow, but not broaden, execution, reuse, inheritance, or automation permissions.

Complete active flows support explicit user/API starts, administrative backfills, one-time schedules, interval schedules, and exact-match platform events. Trigger deliveries are durable and idempotent. They pin the pipeline revision and a least-privilege scope grant, then ask Access to rebuild the owner's current principal before every run. Revoked memberships or reduced permissions block the delivery before source access or output publication. Confidential and restricted events are not accepted through the direct ingress; those require Core's transactional event bridge.

Reusable subflow nodes pin a template reference, version, graph snapshot, and parameter values. Their single input is bound to an explicitly marked inline source inside the snapshot, parameter substitution is data-only, and nesting is bounded. This keeps completed run definitions reproducible even when the source template changes later.

The executable fixtures in fixtures/golden cover the monthly structured-file reconciliation story and the sanctions-screening story with reviewable sample inputs and exact expected outputs.

Development

/mnt/DATA/git/govoplan/.venv/bin/python -m pip install -e .
/mnt/DATA/git/govoplan/.venv/bin/python -m unittest discover -s tests

WebUI:

cd webui
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:$PATH npm run typecheck
npm run test:structure

The implementation epic is govoplan-dataflow#1.