govoplan-dataflow/README.md

GovOPlaN Dataflow

GovOPlaN Dataflow defines and runs governed tabular transformation pipelines. Power users can work with the same pipeline as a graphical node graph or as a constrained SQL query. Every saved definition change produces an immutable revision, and every preview records diagnostics and reproducibility metadata without storing the previewed row contents.

Boundary

• Dataflow: pipeline graphs, revisions, validation, constrained SQL, previews/runs, diagnostics, and lineage.
• Datasources: governed source identity, staging, materializations, frozen states, and bounded source access.
• Connectors: external acquisition, credentials, discovery, and provider health.
• Reporting: governed datasets, analytical views, dashboards, and exports.
• Workflow: orchestration, resumability, approvals, and module handoffs.
• Risk Compliance: sanctions matching policy, review, dispositions, and legal evidence.

Data-subject requests

Dataflow publishes privacy.dsar.dataflow for exact pipeline, revision, reconciliation, run, deployment, trigger, and delivery references and for minimized operator or automation-authority attribution. It never exports graphs, SQL, request/event payloads, reconciliation corrections, authorization snapshots, provenance bodies, errors, source details, hashes, credentials, or output rows. Authoritative input modules locate and correct subject facts; Dataflow does not guess identity by scanning arbitrary transformations.

Exact terminal run and delivery detail can be minimized idempotently, and subject-linked automation authority can be disabled and revoked. Definitions, active work, decisions, deployments, broad pipeline packages, published Datasource outputs, and institutional attribution require review or retention.

Node Library

The canonical backend catalogue is exposed to the WebUI and groups executable nodes by purpose:

Group	Nodes
Load	Inline data, datasource
Combine	Append rows, inner/outer/semi/anti joins
Filter	Column filter, typed-expression filter, remove duplicates
Transform	Select, derive, ordered calculations, typed expression, conversion, replacement, aggregate, partitioned rank, sort, limit, reusable subflow
Quality	Quality rules; keyed reconciliation with stable row identity, explicit before/after evidence, and rerun invalidation hashes
Output	Preview output

Join nodes have explicit left and right ports. Append nodes accept two or more inputs. Derived columns use a constrained operation catalogue rather than arbitrary code. The graph editor validates manual, proximity-created, and reconnected edges against the same port, multiplicity, and cycle rules. SQL round-trips retain node and edge identity, labels, and coordinates while the graph topology remains compatible. Structural changes use a compact layered layout with bounded branch and edge spacing.

Governed inputs are resolved through the versioned Core capability datasources.catalogue; Dataflow imports neither Datasources nor Connectors and stores only opaque datasource references plus expected fingerprints. A source node can request the current, live, or latest frozen state. Fingerprint drift fails visibly instead of silently changing a run.

SQL And Preview Safety

The SQL workbench parses one SELECT or column-aligned UNION BY NAME statement into the canonical graph. The dialect supports projection, aliases, filters, grouping, aggregate functions, sorting, limits, DISTINCT, append, and one two-source equi-join. It rejects DDL, DML, arbitrary subqueries, arbitrary functions, file access, and unchecked pass-through execution. The typed-expression library is a separate allowlisted AST evaluator shared by expression filters and calculated columns. It supports literals, columns, arithmetic, comparisons, boolean logic, CASE, safe casts, predicates, date arithmetic/parts/formatting, and a bounded string/numeric function catalogue without Python evaluation or effectful SQL.

Node definitions, validators, schema propagators, preview executors, and SQL renderers are registered independently in the operator registry. Adding a node no longer requires another branch in the preview or graph-to-SQL dispatch loop. Operators that cannot be represented by constrained SQL declare that explicitly.

Every executable graph also has a versioned typed IR. It preserves graph identity and layout while giving ports, schemas, expressions, parameters, lineage, diagnostics, semantic hashes, and physical results stable contracts. The backend boundary exchanges bounded typed columnar batches that can be serialized as Arrow IPC. The deterministic Python executor remains the default and fallback.

Install .[analytics] to enable the analytical backend. It executes only SQL generated from validated graphs in a separate short-lived DuckDB process. Inputs and outputs cross the process boundary as Arrow IPC; external access, extension installation/loading, persistent secrets, configuration changes, temporary spill files, and multiple DuckDB threads are disabled. Wall-clock, row, byte, memory, file-descriptor, and backend-concurrency limits are applied. Callers can explicitly request reference, duckdb, or auto for a preview.

Preview reads at most 250 rows per source and enforces time, intermediate-row, result-byte, graph-node, and response-row bounds. Saved previews record the pipeline revision, executor version, source fingerprints, node diagnostics, and output summary, but not source or result rows. A preview may return one explicitly requested intermediate node state under the same response bound; that row payload remains ephemeral and is not added to run evidence. The WebUI targets the selected graph node and supports remembered automatic or manual-refresh preview modes.

Saved revisions can also be started through the versioned dataflow.runLifecycle capability or the Run dialog. The first runner is durable and worker-backed: API requests only enqueue idempotent work, while the dataflow.runWorker capability claims leased runs on the dedicated Celery queue. Access rechecks the persisted least-privilege authorization grant before each attempt. Runs expose progress, cancellation, bounded exponential retries, tenant queue/concurrency quotas, notifications, operational metrics, and evidence retention. Expired retention payloads are purged while hashes, row counts, outcomes, and publication references remain as audit evidence.

Development runs may use the bounded reference backend. Staging and production runs require the short-lived isolated DuckDB process. A revision must be promoted from development to staging and then from staging to production before it can run in those environments. Complete results can be published through datasources.publication; publication is rejected when a source or result was truncated. The provider effect is a forward-recovery boundary rather than an atomic database operation. Dataflow records the source and output digests before dispatch, commits a conclusive provider result together with the local run projection, and exposes an outcome_unknown run when acknowledgement is lost. Such a run is never retried until the sink has been reconciled by its stable idempotency key. See docs/DURABLE_RUN_RECOVERY.md.

Governed Definitions And Automation

Pipelines can be complete flows or reusable templates at system, tenant, group, or user scope. Templates never run directly. A permitted consumer derives a new definition that pins the source revision and content hash and records the effective Policy decision and ancestor limits. Inherited definitions remain read-only; lower scopes may narrow, but not broaden, execution, reuse, inheritance, or automation permissions.

Derived definitions report when their source has a newer immutable revision; the source never mutates the child silently. Adopting an update requires the reviewed source revision and hash plus a reason. It appends a new child revision, retains the previous graph and all run evidence, records reviewer and Policy provenance, and returns the child to draft before the changed graph can run or receive automation.

Complete active flows support explicit user/API starts, administrative backfills, one-time schedules, interval schedules, and exact-match platform events. Trigger deliveries are durable and idempotent. They enqueue the same worker-backed run contract, pin the pipeline revision and a least-privilege scope grant, then ask Access to rebuild the owner's current principal before both delivery and execution. Revoked memberships or reduced permissions block the run before source access or output publication. Confidential and restricted events are not accepted through the direct ingress; those require Core's transactional event bridge.

Reusable subflow nodes select a Policy-authorized complete flow or template and an immutable revision. The server resolves the graph instead of accepting a caller-supplied snapshot, records the source hash and Policy decision, and pins closed typed input/output contracts. Their single input is bound to an explicitly marked typed inline source inside the snapshot, parameter substitution is data-only, and cycles across nested references are rejected. Incompatible caller schemas fail validation before execution. This keeps completed run definitions reproducible even when the source definition changes later.

The executable fixtures in fixtures/golden cover monthly structured-file reconciliation, sanctions screening, a HEICO-style current-status export, and the set-based core of a RELE-style booking workflow with reviewable synthetic inputs and exact expected outputs. The detailed source-flow assessment is in docs/HEICO_RELE_ASSESSMENT.md.

The monthly fixture also consumes an explicit decision table. A decision pins the reconciliation key hash and exact input hash plus its reference, actor, time, reason, action, and optional correction. reconcile.decisions annotates matching rows, marks changed-input decisions stale, and reports decisions whose logical row disappeared. It never silently applies a correction to business data; a downstream governed transform or Workflow handoff must interpret the recorded action.

For saved reconciliation pipelines, the preview results provide a review dialog for those rows. Reviewers create a tenant-owned decision set and append accept, reject, correct, or defer decisions with a mandatory reason. Writes use optimistic concurrency; updating a decision creates another immutable revision rather than replacing history. The current projection appears in the ordinary Dataflow source catalogue as dataflow-decision-set:<id> and carries a content fingerprint. A changed input hash is therefore shown as stale and cannot be silently reused. Corrections remain annotations until an explicit downstream transform applies them.

Reporting consumers may either evaluate a pinned pipeline revision or pin one successful published run. An exact run pin is immutable: it cannot be supplied new parameters, and Dataflow reads only the recorded Datasource materialization through the provider-neutral catalogue capability. Both Dataflow run authority and Datasource row access are rechecked for the current principal; the returned lineage retains the run, publication, datasource, materialization, fingerprint, and governance snapshot.

Development

/mnt/DATA/git/govoplan/.venv/bin/python -m pip install -e .
/mnt/DATA/git/govoplan/.venv/bin/python -m unittest discover -s tests

WebUI:

cd webui
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:$PATH npm run typecheck
npm run test:structure

The implementation epic is govoplan-dataflow#1.