112 lines
5.2 KiB
Markdown
112 lines
5.2 KiB
Markdown
# HEICO And RELE Dataflow Assessment
|
|
|
|
This assessment uses the production-shaped SQL files supplied outside the
|
|
repositories:
|
|
|
|
- `heico.sql`: Oracle SQL*Plus query, 285 lines
|
|
- `rele_sql`: PostgreSQL PL/pgSQL program, 4,223 lines
|
|
|
|
The requested `rele.sql` filename does not exist; `rele_sql` is the matching
|
|
input. No production rows or source code are copied into this repository. The
|
|
executable contracts under `fixtures/golden` use synthetic data.
|
|
|
|
## HEICO
|
|
|
|
HEICO is predominantly a declarative relational flow and is a good fit for
|
|
Dataflow. Its relevant operations are:
|
|
|
|
| Source operation | GovOPlaN representation | State |
|
|
| --- | --- | --- |
|
|
| CTE pipeline | Connected graph stages | Modelable; automatic CTE import is missing |
|
|
| Inner and left joins | Join node | Available |
|
|
| `NOT EXISTS` exclusion | Anti join | Available |
|
|
| Latest status per partition | Partitioned row number/rank | Available |
|
|
| Filters, grouping, aggregates | Filter and aggregate nodes | Available |
|
|
| `CASE` and calculated export fields | Ordered calculation block | Available |
|
|
| String/date normalization | Safe expression functions | Available for the used core functions |
|
|
| Correlated scalar lookup | Pre-aggregate plus join | Modelable, but not imported automatically |
|
|
| Oracle package function | Governed source projection/pushdown | Provider-specific work remains |
|
|
| CSV formatting/publication | Reporting or Files output target | Cross-module work remains |
|
|
|
|
`fixtures/golden/heico-student-status` proves the critical shape: select the
|
|
latest status per student, remove excluded records with an anti join, calculate
|
|
conditional/export columns, and project the final result.
|
|
|
|
The complete HEICO query can therefore be rebuilt as a governed graph today.
|
|
What is not yet available is one-click import of its Oracle dialect, automatic
|
|
decomposition of correlated lookups, and an Oracle provider contract for its
|
|
site-specific function.
|
|
|
|
## RELE
|
|
|
|
RELE is not one dataflow. It combines relational transformations with
|
|
procedural orchestration and effects:
|
|
|
|
- temporary raw and staging tables plus indexes
|
|
- repeated ordered `UPDATE ... FROM` enrichment
|
|
- conditional branches and loops
|
|
- validation/error records
|
|
- inserts, updates, and deletes
|
|
- reconciliation and aggregate controls
|
|
- fixed-width and tabular output generation
|
|
|
|
The correct GovOPlaN decomposition is:
|
|
|
|
| Concern | Owning module | Current fit |
|
|
| --- | --- | --- |
|
|
| Immutable raw input and staging states | Datasources | Strong foundation |
|
|
| Set-based normalization, joins, calculations, aggregation | Dataflow | Strong after this slice |
|
|
| Ordered enrichments | Dataflow stages or reusable subflows | Modelable, manual decomposition |
|
|
| Validation and reconciliation | Dataflow quality/reconciliation | Available for single-output checks |
|
|
| User verification, correction, rerun, resumability | Workflow | Foundation exists; flow-specific handoffs remain |
|
|
| Database writes and cleanup | Governed Datasource output/effect capability | Missing explicit effect contract |
|
|
| Warning/error side streams | Multi-output Dataflow nodes | Missing |
|
|
| Fixed-width, CSV, and spreadsheet products | Reporting/Templates/Files | Missing integrated output profile |
|
|
|
|
`fixtures/golden/rele-booking-transform` proves a representative set-based
|
|
slice: ordered calculated columns can normalize types, derive period and
|
|
conditional classifications, calculate signed amounts, validate required
|
|
fields, and aggregate the result.
|
|
|
|
RELE should not be ported as a single opaque stored procedure. Its duplicated
|
|
booking/error paths should become reusable subflows, while Workflow owns the
|
|
run/verify/correct lifecycle. Database effects must remain separately
|
|
authorized and idempotent so a retry cannot repeat an uncertain write.
|
|
|
|
## Added Building Blocks
|
|
|
|
This assessment added:
|
|
|
|
- ordered multi-column calculations, where later expressions can use columns
|
|
calculated earlier in the same block
|
|
- semi and anti joins
|
|
- partitioned `row_number`, `rank`, and `dense_rank`
|
|
- safe `LIKE`, `ILIKE`, `IN`, `BETWEEN`, regular-expression matching and
|
|
replacement
|
|
- padding, splitting, translation, concatenation, extrema, and `NULLIF`
|
|
- date parts, truncation, bounded interval arithmetic/difference, and temporal
|
|
formatting
|
|
- schema propagation, deterministic reference execution, typed IR inventory,
|
|
SQL rendering where representable, and editor controls for each new node
|
|
|
|
The evaluator remains allowlisted. It does not execute arbitrary SQL, Python,
|
|
stored procedures, filesystem access, or network calls.
|
|
|
|
## Remaining Work
|
|
|
|
The next useful slices, in order, are:
|
|
|
|
1. Import and decompose multi-CTE SQL, including window expressions and
|
|
correlated aggregate lookups.
|
|
2. Add reusable lookup/as-of join semantics for temporal master data.
|
|
3. Add multi-output nodes for valid, warning, and rejected rows.
|
|
4. Define explicit idempotent write/effect nodes against Datasource
|
|
capabilities, guarded by Policy and production approvals.
|
|
5. Add parameterized fixed-width/delimited output profiles in Reporting and
|
|
Templates, publishing through Files.
|
|
6. Add provider-side pushdown contracts for source-specific functions without
|
|
exposing unrestricted SQL execution.
|
|
|
|
These gaps remain tracked by
|
|
[`govoplan-dataflow#17`](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/17).
|