72 lines
2.6 KiB
Markdown
72 lines
2.6 KiB
Markdown
# GovOPlaN Dataflow
|
|
|
|
GovOPlaN Dataflow defines and runs governed tabular transformation pipelines.
|
|
Power users can work with the same pipeline as a graphical node graph or as a
|
|
constrained SQL query. Every saved definition change produces an immutable revision, and
|
|
every preview records diagnostics and reproducibility metadata without storing
|
|
the previewed row contents.
|
|
|
|
## Boundary
|
|
|
|
- **Dataflow:** pipeline graphs, revisions, validation, constrained SQL,
|
|
previews/runs, diagnostics, and lineage.
|
|
- **Connectors:** source connections, credentials, discovery, health, and
|
|
bounded source access.
|
|
- **Reporting:** governed datasets, analytical views, dashboards, and exports.
|
|
- **Workflow:** orchestration, resumability, approvals, and module handoffs.
|
|
- **Risk Compliance:** sanctions matching policy, review, dispositions, and
|
|
legal evidence.
|
|
|
|
## Node Library
|
|
|
|
The canonical backend catalogue is exposed to the WebUI and groups executable
|
|
nodes by purpose:
|
|
|
|
| Group | Nodes |
|
|
| --- | --- |
|
|
| Load | Inline data, connector source |
|
|
| Combine | Append rows, join tables |
|
|
| Filter | Filter rows, remove duplicates |
|
|
| Transform | Select columns, derive column, aggregate, sort, limit |
|
|
| Output | Preview output |
|
|
|
|
Join nodes have explicit left and right ports. Append nodes accept two or more
|
|
inputs. Derived columns use a constrained operation catalogue rather than
|
|
arbitrary code.
|
|
|
|
Connector sources are resolved through the versioned Core capability
|
|
`connectors.tabular_sources`; Dataflow does not import Connectors or store its
|
|
credentials. Source references pin an expected fingerprint so schema/content
|
|
drift fails visibly instead of silently changing a run.
|
|
|
|
## SQL And Preview Safety
|
|
|
|
The SQL workbench parses one `SELECT` statement into the canonical graph. The
|
|
dialect supports projection, aliases, filters, grouping, aggregate functions,
|
|
sorting, limits, `DISTINCT`, and one two-source equi-join. It rejects DDL, DML,
|
|
subqueries, arbitrary functions, file access, and unchecked pass-through
|
|
execution.
|
|
|
|
Preview reads at most 250 rows per source and enforces time, intermediate-row,
|
|
result-byte, graph-node, and response-row bounds. Saved previews record the
|
|
pipeline revision, executor version, source fingerprints, node diagnostics,
|
|
and output summary, but not source or result rows.
|
|
|
|
## Development
|
|
|
|
```bash
|
|
/mnt/DATA/git/govoplan/.venv/bin/python -m pip install -e .
|
|
/mnt/DATA/git/govoplan/.venv/bin/python -m unittest discover -s tests
|
|
```
|
|
|
|
WebUI:
|
|
|
|
```bash
|
|
cd webui
|
|
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:$PATH npm run typecheck
|
|
npm run test:structure
|
|
```
|
|
|
|
The implementation epic is
|
|
[`govoplan-dataflow#1`](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/1).
|