govoplan-dataflow/docs/HEICO_RELE_ASSESSMENT.md

HEICO And RELE Dataflow Assessment

This assessment uses the production-shaped SQL files supplied outside the repositories:

• heico.sql: Oracle SQL*Plus query, 285 lines
• rele_sql: PostgreSQL PL/pgSQL program, 4,223 lines

The requested rele.sql filename does not exist; rele_sql is the matching input. No production rows or source code are copied into this repository. The executable contracts under fixtures/golden use synthetic data.

HEICO

HEICO is predominantly a declarative relational flow and is a good fit for Dataflow. Its relevant operations are:

Source operation	GovOPlaN representation	State
CTE pipeline	Connected graph stages	Modelable; automatic CTE import is missing
Inner and left joins	Join node	Available
NOT EXISTS exclusion	Anti join	Available
Latest status per partition	Partitioned row number/rank	Available
Filters, grouping, aggregates	Filter and aggregate nodes	Available
CASE and calculated export fields	Ordered calculation block	Available
String/date normalization	Safe expression functions	Available for the used core functions
Correlated scalar lookup	Pre-aggregate plus join	Modelable, but not imported automatically
Oracle package function	Governed source projection/pushdown	Provider-specific work remains
CSV formatting/publication	Reporting or Files output target	Cross-module work remains

fixtures/golden/heico-student-status proves the critical shape: select the latest status per student, remove excluded records with an anti join, calculate conditional/export columns, and project the final result.

The complete HEICO query can therefore be rebuilt as a governed graph today. What is not yet available is one-click import of its Oracle dialect, automatic decomposition of correlated lookups, and an Oracle provider contract for its site-specific function.

RELE

RELE is not one dataflow. It combines relational transformations with procedural orchestration and effects:

• temporary raw and staging tables plus indexes
• repeated ordered UPDATE ... FROM enrichment
• conditional branches and loops
• validation/error records
• inserts, updates, and deletes
• reconciliation and aggregate controls
• fixed-width and tabular output generation

The correct GovOPlaN decomposition is:

Concern	Owning module	Current fit
Immutable raw input and staging states	Datasources	Strong foundation
Set-based normalization, joins, calculations, aggregation	Dataflow	Strong after this slice
Ordered enrichments	Dataflow stages or reusable subflows	Modelable, manual decomposition
Validation and reconciliation	Dataflow quality/reconciliation	Available for single-output checks
User verification, correction, rerun, resumability	Workflow	Foundation exists; flow-specific handoffs remain
Database writes and cleanup	Governed Datasource output/effect capability	Missing explicit effect contract
Warning/error side streams	Multi-output Dataflow nodes	Missing
Fixed-width, CSV, and spreadsheet products	Reporting/Templates/Files	Missing integrated output profile

fixtures/golden/rele-booking-transform proves a representative set-based slice: ordered calculated columns can normalize types, derive period and conditional classifications, calculate signed amounts, validate required fields, and aggregate the result.

RELE should not be ported as a single opaque stored procedure. Its duplicated booking/error paths should become reusable subflows, while Workflow owns the run/verify/correct lifecycle. Database effects must remain separately authorized and idempotent so a retry cannot repeat an uncertain write.

Added Building Blocks

This assessment added:

• ordered multi-column calculations, where later expressions can use columns calculated earlier in the same block
• semi and anti joins
• partitioned row_number, rank, and dense_rank
• safe LIKE, ILIKE, IN, BETWEEN, regular-expression matching and replacement
• padding, splitting, translation, concatenation, extrema, and NULLIF
• date parts, truncation, bounded interval arithmetic/difference, and temporal formatting
• schema propagation, deterministic reference execution, typed IR inventory, SQL rendering where representable, and editor controls for each new node

The evaluator remains allowlisted. It does not execute arbitrary SQL, Python, stored procedures, filesystem access, or network calls.

Remaining Work

The next useful slices, in order, are:

1. Import and decompose multi-CTE SQL, including window expressions and correlated aggregate lookups.
2. Add reusable lookup/as-of join semantics for temporal master data.
3. Add multi-output nodes for valid, warning, and rejected rows.
4. Define explicit idempotent write/effect nodes against Datasource capabilities, guarded by Policy and production approvals.
5. Add parameterized fixed-width/delimited output profiles in Reporting and Templates, publishing through Files.
6. Add provider-side pushdown contracts for source-specific functions without exposing unrestricted SQL execution.

These gaps remain tracked by govoplan-dataflow#17.