govoplan-dataflow/README.md

# GovOPlaN Dataflow

GovOPlaN Dataflow defines and runs governed tabular transformation pipelines.
Power users can work with the same pipeline as a graphical node graph or as a
constrained SQL query. Every saved definition change produces an immutable
revision, and every preview records diagnostics and reproducibility metadata
without storing the previewed row contents.

## Boundary

- **Dataflow:** pipeline graphs, revisions, validation, constrained SQL,
  previews/runs, diagnostics, and lineage.
- **Datasources:** governed source identity, staging, materializations, frozen
  states, and bounded source access.
- **Connectors:** external acquisition, credentials, discovery, and provider
  health.
- **Reporting:** governed datasets, analytical views, dashboards, and exports.
- **Workflow:** orchestration, resumability, approvals, and module handoffs.
- **Risk Compliance:** sanctions matching policy, review, dispositions, and
  legal evidence.

## Data-subject requests

Dataflow publishes `privacy.dsar.dataflow` for exact pipeline, revision,
reconciliation, run, deployment, trigger, and delivery references and for
minimized operator or automation-authority attribution. It never exports
graphs, SQL, request/event payloads, reconciliation corrections, authorization
snapshots, provenance bodies, errors, source details, hashes, credentials, or
output rows. Authoritative input modules locate and correct subject facts;
Dataflow does not guess identity by scanning arbitrary transformations.

Exact terminal run and delivery detail can be minimized idempotently, and
subject-linked automation authority can be disabled and revoked. Definitions,
active work, decisions, deployments, broad pipeline packages, published
Datasource outputs, and institutional attribution require review or retention.

## Node Library

The canonical backend catalogue is exposed to the WebUI and groups executable
nodes by purpose:

| Group | Nodes |
| --- | --- |
| Load | Inline data, datasource |
| Combine | Append rows, inner/outer/semi/anti joins |
| Filter | Column filter, typed-expression filter, remove duplicates |
| Transform | Select, derive, ordered calculations, typed expression, conversion, replacement, aggregate, partitioned rank, sort, limit, reusable subflow |
| Quality | Quality rules; keyed reconciliation with stable row identity, explicit before/after evidence, and rerun invalidation hashes |
| Output | Preview output |

Join nodes have explicit left and right ports. Append nodes accept two or more
inputs. Derived columns use a constrained operation catalogue rather than
arbitrary code. The graph editor validates manual, proximity-created, and
reconnected edges against the same port, multiplicity, and cycle rules.
SQL round-trips retain node and edge identity, labels, and coordinates while
the graph topology remains compatible. Structural changes use a compact
layered layout with bounded branch and edge spacing.

Governed inputs are resolved through the versioned Core capability
`datasources.catalogue`; Dataflow imports neither Datasources nor Connectors and
stores only opaque datasource references plus expected fingerprints. A source
node can request the current, live, or latest frozen state. Fingerprint drift
fails visibly instead of silently changing a run.

## SQL And Preview Safety

The SQL workbench parses one `SELECT` or column-aligned `UNION BY NAME`
statement into the canonical graph. The dialect supports projection, aliases,
filters, grouping, aggregate functions, sorting, limits, `DISTINCT`, append,
and one two-source equi-join. It rejects DDL, DML, arbitrary subqueries,
arbitrary functions, file access, and unchecked pass-through execution.
The typed-expression library is a separate allowlisted AST evaluator shared by
expression filters and calculated columns. It supports literals, columns,
arithmetic, comparisons, boolean logic, `CASE`, safe casts, predicates,
date arithmetic/parts/formatting, and a bounded string/numeric function
catalogue without Python evaluation or effectful SQL.

Node definitions, validators, schema propagators, preview executors, and SQL
renderers are registered independently in the operator registry. Adding a node
no longer requires another branch in the preview or graph-to-SQL dispatch loop.
Operators that cannot be represented by constrained SQL declare that
explicitly.

Every executable graph also has a versioned typed IR. It preserves graph
identity and layout while giving ports, schemas, expressions, parameters,
lineage, diagnostics, semantic hashes, and physical results stable contracts.
The backend boundary exchanges bounded typed columnar batches that can be
serialized as Arrow IPC. The deterministic Python executor remains the default
and fallback.

Install `.[analytics]` to enable the analytical backend. It executes only SQL
generated from validated graphs in a separate short-lived DuckDB process.
Inputs and outputs cross the process boundary as Arrow IPC; external access,
extension installation/loading, persistent secrets, configuration changes,
temporary spill files, and multiple DuckDB threads are disabled. Wall-clock,
row, byte, memory, file-descriptor, and backend-concurrency limits are applied.
Callers can explicitly request `reference`, `duckdb`, or `auto` for a preview.

Preview reads at most 250 rows per source and enforces time, intermediate-row,
result-byte, graph-node, and response-row bounds. Saved previews record the
pipeline revision, executor version, source fingerprints, node diagnostics,
and output summary, but not source or result rows. A preview may return one
explicitly requested intermediate node state under the same response bound;
that row payload remains ephemeral and is not added to run evidence. The WebUI
targets the selected graph node and supports remembered automatic or
manual-refresh preview modes.

Saved revisions can also be started through the versioned
`dataflow.runLifecycle` capability or the Run dialog. The first runner is
durable and worker-backed: API requests only enqueue idempotent work, while
the `dataflow.runWorker` capability claims leased runs on the dedicated Celery
queue. Access rechecks the persisted least-privilege authorization grant before
each attempt. Runs expose progress, cancellation, bounded exponential retries,
tenant queue/concurrency quotas, notifications, operational metrics, and
evidence retention. Expired retention payloads are purged while hashes, row
counts, outcomes, and publication references remain as audit evidence.

Development runs may use the bounded reference backend. Staging and production
runs require the short-lived isolated DuckDB process. A revision must be
promoted from development to staging and then from staging to production before
it can run in those environments. Complete results can be published through
`datasources.publication`; publication is rejected when a source or result was
truncated. The provider effect is a forward-recovery boundary rather than an
atomic database operation. Dataflow records the source and output digests before
dispatch, commits a conclusive provider result together with the local run
projection, and exposes an `outcome_unknown` run when acknowledgement is lost.
Such a run is never retried until the sink has been reconciled by its stable
idempotency key. See
[`docs/DURABLE_RUN_RECOVERY.md`](docs/DURABLE_RUN_RECOVERY.md).

## Governed Definitions And Automation

Pipelines can be complete flows or reusable templates at system, tenant,
group, or user scope. Templates never run directly. A permitted consumer
derives a new definition that pins the source revision and content hash and
records the effective Policy decision and ancestor limits. Inherited
definitions remain read-only; lower scopes may narrow, but not broaden,
execution, reuse, inheritance, or automation permissions.

Derived definitions report when their source has a newer immutable revision;
the source never mutates the child silently. Adopting an update requires the
reviewed source revision and hash plus a reason. It appends a new child
revision, retains the previous graph and all run evidence, records reviewer
and Policy provenance, and returns the child to draft before the changed graph
can run or receive automation.

Complete active flows support explicit user/API starts, administrative
backfills, one-time schedules, interval schedules, and exact-match platform
events. Trigger deliveries are durable and idempotent. They enqueue the same
worker-backed run contract, pin the pipeline revision and a least-privilege
scope grant, then ask Access to rebuild the owner's current principal before
both delivery and execution. Revoked memberships or reduced permissions block
the run before source access or output publication.
Confidential and restricted events are not accepted through the direct
ingress; those require Core's transactional event bridge.

Reusable subflow nodes select a Policy-authorized complete flow or template and
an immutable revision. The server resolves the graph instead of accepting a
caller-supplied snapshot, records the source hash and Policy decision, and pins
closed typed input/output contracts. Their single input is bound to an
explicitly marked typed inline source inside the snapshot, parameter
substitution is data-only, and cycles across nested references are rejected.
Incompatible caller schemas fail validation before execution. This keeps
completed run definitions reproducible even when the source definition changes
later.

The executable fixtures in `fixtures/golden` cover monthly structured-file
reconciliation, sanctions screening, a HEICO-style current-status export, and
the set-based core of a RELE-style booking workflow with reviewable synthetic
inputs and exact expected outputs. The detailed source-flow assessment is in
[`docs/HEICO_RELE_ASSESSMENT.md`](docs/HEICO_RELE_ASSESSMENT.md).

The monthly fixture also consumes an explicit decision table. A decision pins
the reconciliation key hash and exact input hash plus its reference, actor,
time, reason, action, and optional correction. `reconcile.decisions` annotates
matching rows, marks changed-input decisions stale, and reports decisions whose
logical row disappeared. It never silently applies a correction to business
data; a downstream governed transform or Workflow handoff must interpret the
recorded action.

For saved reconciliation pipelines, the preview results provide a review
dialog for those rows. Reviewers create a tenant-owned decision set and append
accept, reject, correct, or defer decisions with a mandatory reason. Writes use
optimistic concurrency; updating a decision creates another immutable revision
rather than replacing history. The current projection appears in the ordinary
Dataflow source catalogue as `dataflow-decision-set:<id>` and carries a content
fingerprint. A changed input hash is therefore shown as stale and cannot be
silently reused. Corrections remain annotations until an explicit downstream
transform applies them.

Reporting consumers may either evaluate a pinned pipeline revision or pin one
successful published run. An exact run pin is immutable: it cannot be supplied
new parameters, and Dataflow reads only the recorded Datasource materialization
through the provider-neutral catalogue capability. Both Dataflow run authority
and Datasource row access are rechecked for the current principal; the returned
lineage retains the run, publication, datasource, materialization, fingerprint,
and governance snapshot.

## Development

```bash
/mnt/DATA/git/govoplan/.venv/bin/python -m pip install -e .
/mnt/DATA/git/govoplan/.venv/bin/python -m unittest discover -s tests
```

WebUI:

```bash
cd webui
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:$PATH npm run typecheck
npm run test:structure
```

The implementation epic is
[`govoplan-dataflow#1`](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/1).