145 lines
7.1 KiB
Markdown
145 lines
7.1 KiB
Markdown
# GovOPlaN Dataflow
|
|
|
|
GovOPlaN Dataflow defines and runs governed tabular transformation pipelines.
|
|
Power users can work with the same pipeline as a graphical node graph or as a
|
|
constrained SQL query. Every saved definition change produces an immutable
|
|
revision, and every preview records diagnostics and reproducibility metadata
|
|
without storing the previewed row contents.
|
|
|
|
## Boundary
|
|
|
|
- **Dataflow:** pipeline graphs, revisions, validation, constrained SQL,
|
|
previews/runs, diagnostics, and lineage.
|
|
- **Datasources:** governed source identity, staging, materializations, frozen
|
|
states, and bounded source access.
|
|
- **Connectors:** external acquisition, credentials, discovery, and provider
|
|
health.
|
|
- **Reporting:** governed datasets, analytical views, dashboards, and exports.
|
|
- **Workflow:** orchestration, resumability, approvals, and module handoffs.
|
|
- **Risk Compliance:** sanctions matching policy, review, dispositions, and
|
|
legal evidence.
|
|
|
|
## Node Library
|
|
|
|
The canonical backend catalogue is exposed to the WebUI and groups executable
|
|
nodes by purpose:
|
|
|
|
| Group | Nodes |
|
|
| --- | --- |
|
|
| Load | Inline data, datasource |
|
|
| Combine | Append rows, join tables |
|
|
| Filter | Column filter, typed-expression filter, remove duplicates |
|
|
| Transform | Select, derive, typed expression, conversion, replacement, aggregate, sort, limit, reusable subflow |
|
|
| Quality | Quality rules, keyed reconciliation |
|
|
| Output | Preview output |
|
|
|
|
Join nodes have explicit left and right ports. Append nodes accept two or more
|
|
inputs. Derived columns use a constrained operation catalogue rather than
|
|
arbitrary code. The graph editor validates manual, proximity-created, and
|
|
reconnected edges against the same port, multiplicity, and cycle rules.
|
|
SQL round-trips retain node and edge identity, labels, and coordinates while
|
|
the graph topology remains compatible. Structural changes use a compact
|
|
layered layout with bounded branch and edge spacing.
|
|
|
|
Governed inputs are resolved through the versioned Core capability
|
|
`datasources.catalogue`; Dataflow imports neither Datasources nor Connectors and
|
|
stores only opaque datasource references plus expected fingerprints. A source
|
|
node can request the current, live, or latest frozen state. Fingerprint drift
|
|
fails visibly instead of silently changing a run.
|
|
|
|
## SQL And Preview Safety
|
|
|
|
The SQL workbench parses one `SELECT` or column-aligned `UNION BY NAME`
|
|
statement into the canonical graph. The dialect supports projection, aliases,
|
|
filters, grouping, aggregate functions, sorting, limits, `DISTINCT`, append,
|
|
and one two-source equi-join. It rejects DDL, DML, arbitrary subqueries,
|
|
arbitrary functions, file access, and unchecked pass-through execution.
|
|
The typed-expression library is a separate allowlisted AST evaluator shared by
|
|
expression filters and calculated columns. It supports literals, columns,
|
|
arithmetic, comparisons, boolean logic, `CASE`, safe casts, and a bounded
|
|
string/numeric function catalogue without Python evaluation or effectful SQL.
|
|
|
|
Node definitions, validators, schema propagators, preview executors, and SQL
|
|
renderers are registered independently in the operator registry. Adding a node
|
|
no longer requires another branch in the preview or graph-to-SQL dispatch loop.
|
|
Operators that cannot be represented by constrained SQL declare that
|
|
explicitly.
|
|
|
|
Every executable graph also has a versioned typed IR. It preserves graph
|
|
identity and layout while giving ports, schemas, expressions, parameters,
|
|
lineage, diagnostics, semantic hashes, and physical results stable contracts.
|
|
The backend boundary exchanges bounded typed columnar batches that can be
|
|
serialized as Arrow IPC. The deterministic Python executor remains the default
|
|
and fallback.
|
|
|
|
Install `.[analytics]` to enable the analytical backend. It executes only SQL
|
|
generated from validated graphs in a separate short-lived DuckDB process.
|
|
Inputs and outputs cross the process boundary as Arrow IPC; external access,
|
|
extension installation/loading, persistent secrets, configuration changes,
|
|
temporary spill files, and multiple DuckDB threads are disabled. Wall-clock,
|
|
row, byte, memory, file-descriptor, and backend-concurrency limits are applied.
|
|
Callers can explicitly request `reference`, `duckdb`, or `auto` for a preview.
|
|
|
|
Preview reads at most 250 rows per source and enforces time, intermediate-row,
|
|
result-byte, graph-node, and response-row bounds. Saved previews record the
|
|
pipeline revision, executor version, source fingerprints, node diagnostics,
|
|
and output summary, but not source or result rows. A preview may return one
|
|
explicitly requested intermediate node state under the same response bound;
|
|
that row payload remains ephemeral and is not added to run evidence. The WebUI
|
|
targets the selected graph node and supports remembered automatic or
|
|
manual-refresh preview modes.
|
|
|
|
Saved revisions can also be started through the versioned
|
|
`dataflow.runLifecycle` capability or the Run dialog. The first runner is
|
|
synchronous and bounded. It records an idempotent terminal run, pins the
|
|
pipeline revision, and can atomically publish a complete result through
|
|
`datasources.publication`. Publication is rejected when a source or result was
|
|
truncated; larger asynchronous and artifact-backed execution belongs to a
|
|
subsequent runner slice.
|
|
|
|
## Governed Definitions And Automation
|
|
|
|
Pipelines can be complete flows or reusable templates at system, tenant,
|
|
group, or user scope. Templates never run directly. A permitted consumer
|
|
derives a new definition that pins the source revision and content hash and
|
|
records the effective Policy decision and ancestor limits. Inherited
|
|
definitions remain read-only; lower scopes may narrow, but not broaden,
|
|
execution, reuse, inheritance, or automation permissions.
|
|
|
|
Complete active flows support explicit user/API starts, administrative
|
|
backfills, one-time schedules, interval schedules, and exact-match platform
|
|
events. Trigger deliveries are durable and idempotent. They pin the pipeline
|
|
revision and a least-privilege scope grant, then ask Access to rebuild the
|
|
owner's current principal before every run. Revoked memberships or reduced
|
|
permissions block the delivery before source access or output publication.
|
|
Confidential and restricted events are not accepted through the direct
|
|
ingress; those require Core's transactional event bridge.
|
|
|
|
Reusable subflow nodes pin a template reference, version, graph snapshot, and
|
|
parameter values. Their single input is bound to an explicitly marked inline
|
|
source inside the snapshot, parameter substitution is data-only, and nesting
|
|
is bounded. This keeps completed run definitions reproducible even when the
|
|
source template changes later.
|
|
|
|
The executable fixtures in `fixtures/golden` cover the monthly structured-file
|
|
reconciliation story and the sanctions-screening story with reviewable sample
|
|
inputs and exact expected outputs.
|
|
|
|
## Development
|
|
|
|
```bash
|
|
/mnt/DATA/git/govoplan/.venv/bin/python -m pip install -e .
|
|
/mnt/DATA/git/govoplan/.venv/bin/python -m unittest discover -s tests
|
|
```
|
|
|
|
WebUI:
|
|
|
|
```bash
|
|
cd webui
|
|
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:$PATH npm run typecheck
|
|
npm run test:structure
|
|
```
|
|
|
|
The implementation epic is
|
|
[`govoplan-dataflow#1`](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/1).
|