GovOPlaN/govoplan-dataflow
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(dataflow): model production SQL flow patterns

Browse Source
This commit is contained in:
zemion
2026-07-30 18:19:18 +02:00
parent 0664570607
commit e091042948
26 changed files with 2169 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+10 -7
View File
@@ -27,9 +27,9 @@ nodes by purpose:
| Group | Nodes |
| --- | --- |
| Load | Inline data, datasource |
| Combine | Append rows, join tables |
| Combine | Append rows, inner/outer/semi/anti joins |
| Filter | Column filter, typed-expression filter, remove duplicates |
| Transform | Select, derive, typed expression, conversion, replacement, aggregate, sort, limit, reusable subflow |
| Transform | Select, derive, ordered calculations, typed expression, conversion, replacement, aggregate, partitioned rank, sort, limit, reusable subflow |
| Quality | Quality rules, keyed reconciliation |
| Output | Preview output |
@@ -56,8 +56,9 @@ and one two-source equi-join. It rejects DDL, DML, arbitrary subqueries,
arbitrary functions, file access, and unchecked pass-through execution.
The typed-expression library is a separate allowlisted AST evaluator shared by
expression filters and calculated columns. It supports literals, columns,
arithmetic, comparisons, boolean logic, `CASE`, safe casts, and a bounded
string/numeric function catalogue without Python evaluation or effectful SQL.
arithmetic, comparisons, boolean logic, `CASE`, safe casts, predicates,
date arithmetic/parts/formatting, and a bounded string/numeric function
catalogue without Python evaluation or effectful SQL.
Node definitions, validators, schema propagators, preview executors, and SQL
renderers are registered independently in the operator registry. Adding a node
@@ -131,9 +132,11 @@ source inside the snapshot, parameter substitution is data-only, and nesting
is bounded. This keeps completed run definitions reproducible even when the
source template changes later.
The executable fixtures in `fixtures/golden` cover the monthly structured-file
reconciliation story and the sanctions-screening story with reviewable sample
inputs and exact expected outputs.
The executable fixtures in `fixtures/golden` cover monthly structured-file
reconciliation, sanctions screening, a HEICO-style current-status export, and
the set-based core of a RELE-style booking workflow with reviewable synthetic
inputs and exact expected outputs. The detailed source-flow assessment is in
[`docs/HEICO_RELE_ASSESSMENT.md`](docs/HEICO_RELE_ASSESSMENT.md).
## Development