feat(dataflow): model production SQL flow patterns
This commit is contained in:
@@ -0,0 +1,111 @@
|
||||
# HEICO And RELE Dataflow Assessment
|
||||
|
||||
This assessment uses the production-shaped SQL files supplied outside the
|
||||
repositories:
|
||||
|
||||
- `heico.sql`: Oracle SQL*Plus query, 285 lines
|
||||
- `rele_sql`: PostgreSQL PL/pgSQL program, 4,223 lines
|
||||
|
||||
The requested `rele.sql` filename does not exist; `rele_sql` is the matching
|
||||
input. No production rows or source code are copied into this repository. The
|
||||
executable contracts under `fixtures/golden` use synthetic data.
|
||||
|
||||
## HEICO
|
||||
|
||||
HEICO is predominantly a declarative relational flow and is a good fit for
|
||||
Dataflow. Its relevant operations are:
|
||||
|
||||
| Source operation | GovOPlaN representation | State |
|
||||
| --- | --- | --- |
|
||||
| CTE pipeline | Connected graph stages | Modelable; automatic CTE import is missing |
|
||||
| Inner and left joins | Join node | Available |
|
||||
| `NOT EXISTS` exclusion | Anti join | Available |
|
||||
| Latest status per partition | Partitioned row number/rank | Available |
|
||||
| Filters, grouping, aggregates | Filter and aggregate nodes | Available |
|
||||
| `CASE` and calculated export fields | Ordered calculation block | Available |
|
||||
| String/date normalization | Safe expression functions | Available for the used core functions |
|
||||
| Correlated scalar lookup | Pre-aggregate plus join | Modelable, but not imported automatically |
|
||||
| Oracle package function | Governed source projection/pushdown | Provider-specific work remains |
|
||||
| CSV formatting/publication | Reporting or Files output target | Cross-module work remains |
|
||||
|
||||
`fixtures/golden/heico-student-status` proves the critical shape: select the
|
||||
latest status per student, remove excluded records with an anti join, calculate
|
||||
conditional/export columns, and project the final result.
|
||||
|
||||
The complete HEICO query can therefore be rebuilt as a governed graph today.
|
||||
What is not yet available is one-click import of its Oracle dialect, automatic
|
||||
decomposition of correlated lookups, and an Oracle provider contract for its
|
||||
site-specific function.
|
||||
|
||||
## RELE
|
||||
|
||||
RELE is not one dataflow. It combines relational transformations with
|
||||
procedural orchestration and effects:
|
||||
|
||||
- temporary raw and staging tables plus indexes
|
||||
- repeated ordered `UPDATE ... FROM` enrichment
|
||||
- conditional branches and loops
|
||||
- validation/error records
|
||||
- inserts, updates, and deletes
|
||||
- reconciliation and aggregate controls
|
||||
- fixed-width and tabular output generation
|
||||
|
||||
The correct GovOPlaN decomposition is:
|
||||
|
||||
| Concern | Owning module | Current fit |
|
||||
| --- | --- | --- |
|
||||
| Immutable raw input and staging states | Datasources | Strong foundation |
|
||||
| Set-based normalization, joins, calculations, aggregation | Dataflow | Strong after this slice |
|
||||
| Ordered enrichments | Dataflow stages or reusable subflows | Modelable, manual decomposition |
|
||||
| Validation and reconciliation | Dataflow quality/reconciliation | Available for single-output checks |
|
||||
| User verification, correction, rerun, resumability | Workflow | Foundation exists; flow-specific handoffs remain |
|
||||
| Database writes and cleanup | Governed Datasource output/effect capability | Missing explicit effect contract |
|
||||
| Warning/error side streams | Multi-output Dataflow nodes | Missing |
|
||||
| Fixed-width, CSV, and spreadsheet products | Reporting/Templates/Files | Missing integrated output profile |
|
||||
|
||||
`fixtures/golden/rele-booking-transform` proves a representative set-based
|
||||
slice: ordered calculated columns can normalize types, derive period and
|
||||
conditional classifications, calculate signed amounts, validate required
|
||||
fields, and aggregate the result.
|
||||
|
||||
RELE should not be ported as a single opaque stored procedure. Its duplicated
|
||||
booking/error paths should become reusable subflows, while Workflow owns the
|
||||
run/verify/correct lifecycle. Database effects must remain separately
|
||||
authorized and idempotent so a retry cannot repeat an uncertain write.
|
||||
|
||||
## Added Building Blocks
|
||||
|
||||
This assessment added:
|
||||
|
||||
- ordered multi-column calculations, where later expressions can use columns
|
||||
calculated earlier in the same block
|
||||
- semi and anti joins
|
||||
- partitioned `row_number`, `rank`, and `dense_rank`
|
||||
- safe `LIKE`, `ILIKE`, `IN`, `BETWEEN`, regular-expression matching and
|
||||
replacement
|
||||
- padding, splitting, translation, concatenation, extrema, and `NULLIF`
|
||||
- date parts, truncation, bounded interval arithmetic/difference, and temporal
|
||||
formatting
|
||||
- schema propagation, deterministic reference execution, typed IR inventory,
|
||||
SQL rendering where representable, and editor controls for each new node
|
||||
|
||||
The evaluator remains allowlisted. It does not execute arbitrary SQL, Python,
|
||||
stored procedures, filesystem access, or network calls.
|
||||
|
||||
## Remaining Work
|
||||
|
||||
The next useful slices, in order, are:
|
||||
|
||||
1. Import and decompose multi-CTE SQL, including window expressions and
|
||||
correlated aggregate lookups.
|
||||
2. Add reusable lookup/as-of join semantics for temporal master data.
|
||||
3. Add multi-output nodes for valid, warning, and rejected rows.
|
||||
4. Define explicit idempotent write/effect nodes against Datasource
|
||||
capabilities, guarded by Policy and production approvals.
|
||||
5. Add parameterized fixed-width/delimited output profiles in Reporting and
|
||||
Templates, publishing through Files.
|
||||
6. Add provider-side pushdown contracts for source-specific functions without
|
||||
exposing unrestricted SQL execution.
|
||||
|
||||
These gaps remain tracked by
|
||||
[`govoplan-dataflow#17`](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/17).
|
||||
Reference in New Issue
Block a user