# GovOPlaN Dataflow

GovOPlaN Dataflow defines and runs governed tabular transformation pipelines.
Power users can work with the same pipeline as a graphical node graph or as a
constrained SQL query. Every saved definition change produces an immutable revision, and
every preview records diagnostics and reproducibility metadata without storing
the previewed row contents.

## Boundary

- **Dataflow:** pipeline graphs, revisions, validation, constrained SQL,
  previews/runs, diagnostics, and lineage.
- **Connectors:** source connections, credentials, discovery, health, and
  bounded source access.
- **Reporting:** governed datasets, analytical views, dashboards, and exports.
- **Workflow:** orchestration, resumability, approvals, and module handoffs.
- **Risk Compliance:** sanctions matching policy, review, dispositions, and
  legal evidence.

## Node Library

The canonical backend catalogue is exposed to the WebUI and groups executable
nodes by purpose:

| Group | Nodes |
| --- | --- |
| Load | Inline data, connector source |
| Combine | Append rows, join tables |
| Filter | Filter rows, remove duplicates |
| Transform | Select columns, derive column, aggregate, sort, limit |
| Output | Preview output |

Join nodes have explicit left and right ports. Append nodes accept two or more
inputs. Derived columns use a constrained operation catalogue rather than
arbitrary code.

Connector sources are resolved through the versioned Core capability
`connectors.tabular_sources`; Dataflow does not import Connectors or store its
credentials. Source references pin an expected fingerprint so schema/content
drift fails visibly instead of silently changing a run.

## SQL And Preview Safety

The SQL workbench parses one `SELECT` statement into the canonical graph. The
dialect supports projection, aliases, filters, grouping, aggregate functions,
sorting, limits, `DISTINCT`, and one two-source equi-join. It rejects DDL, DML,
subqueries, arbitrary functions, file access, and unchecked pass-through
execution.

Preview reads at most 250 rows per source and enforces time, intermediate-row,
result-byte, graph-node, and response-row bounds. Saved previews record the
pipeline revision, executor version, source fingerprints, node diagnostics,
and output summary, but not source or result rows.

## Development

```bash
/mnt/DATA/git/govoplan/.venv/bin/python -m pip install -e .
/mnt/DATA/git/govoplan/.venv/bin/python -m unittest discover -s tests
```

WebUI:

```bash
cd webui
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:$PATH npm run typecheck
npm run test:structure
```

The implementation epic is
[`govoplan-dataflow#1`](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/1).