# GovOPlaN Dataflow

GovOPlaN Dataflow defines and runs governed tabular transformation pipelines.
Power users can work with the same pipeline as a graphical node graph or as a
constrained SQL query. Every saved definition change produces an immutable
revision, and every preview records diagnostics and reproducibility metadata
without storing the previewed row contents.

## Boundary

- **Dataflow:** pipeline graphs, revisions, validation, constrained SQL,
  previews/runs, diagnostics, and lineage.
- **Datasources:** governed source identity, staging, materializations, frozen
  states, and bounded source access.
- **Connectors:** external acquisition, credentials, discovery, and provider
  health.
- **Reporting:** governed datasets, analytical views, dashboards, and exports.
- **Workflow:** orchestration, resumability, approvals, and module handoffs.
- **Risk Compliance:** sanctions matching policy, review, dispositions, and
  legal evidence.

## Node Library

The canonical backend catalogue is exposed to the WebUI and groups executable
nodes by purpose:

| Group | Nodes |
| --- | --- |
| Load | Inline data, datasource |
| Combine | Append rows, join tables |
| Filter | Filter rows, remove duplicates |
| Transform | Select columns, derive column, aggregate, sort, limit |
| Output | Preview output |

Join nodes have explicit left and right ports. Append nodes accept two or more
inputs. Derived columns use a constrained operation catalogue rather than
arbitrary code.

Governed inputs are resolved through the versioned Core capability
`datasources.catalogue`; Dataflow imports neither Datasources nor Connectors and
stores only opaque datasource references plus expected fingerprints. A source
node can request the current, live, or latest frozen state. Fingerprint drift
fails visibly instead of silently changing a run.

## SQL And Preview Safety

The SQL workbench parses one `SELECT` statement into the canonical graph. The
dialect supports projection, aliases, filters, grouping, aggregate functions,
sorting, limits, `DISTINCT`, and one two-source equi-join. It rejects DDL, DML,
subqueries, arbitrary functions, file access, and unchecked pass-through
execution.

Preview reads at most 250 rows per source and enforces time, intermediate-row,
result-byte, graph-node, and response-row bounds. Saved previews record the
pipeline revision, executor version, source fingerprints, node diagnostics,
and output summary, but not source or result rows.

Saved revisions can also be started through the versioned
`dataflow.runLifecycle` capability or the Run dialog. The first runner is
synchronous and bounded. It records an idempotent terminal run, pins the
pipeline revision, and can atomically publish a complete result through
`datasources.publication`. Publication is rejected when a source or result was
truncated; larger asynchronous and artifact-backed execution belongs to a
subsequent runner slice.

## Development

```bash
/mnt/DATA/git/govoplan/.venv/bin/python -m pip install -e .
/mnt/DATA/git/govoplan/.venv/bin/python -m unittest discover -s tests
```

WebUI:

```bash
cd webui
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:$PATH npm run typecheck
npm run test:structure
```

The implementation epic is
[`govoplan-dataflow#1`](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/1).