[Feature] Validate HEICO and RELE production flows with executable golden contracts #17

New Issue

2026-07-30T17:58:49+02:00

zemion commented

2026-07-30 17:58:49 +02:00

Context

Two production SQL flows were reviewed as concrete Dataflow acceptance cases:

Products/govoplan/heico.sql: an Oracle CTE pipeline for student status selection and CSV output.
Products/govoplan/rele_sql: a PostgreSQL/PLpgSQL personnel-cost transformation and booking/report export pipeline. The supplied rele.sql path currently exists as rele_sql.

HEICO is predominantly a declarative data graph. RELE combines set-based transformations with staging, validation/error streams, procedural branching, ordered mutation, grouped report generation, and formatted outputs; the latter must therefore be decomposed across Dataflow, Workflow, Datasources, Reporting, and Files rather than represented as one opaque SQL node.

No production rows or third-party implementation code should be committed. Use small synthetic fixtures that preserve the semantics.

Immediate slice

Add an ordered multi-column calculation block for calculated and conditional columns.
Extend the safe expression profile for the date/text/predicate functions required by the fixtures.
Add semi/anti join semantics for EXISTS/NOT EXISTS flows.
Add partitioned row-number/rank semantics for latest/first-per-group selection.
Add executable HEICO-shaped and RELE-shaped golden fixtures with exact expected outputs.
Record the coverage matrix and decomposition boundary in repository documentation.

Follow-up scope

SQL/CTE and procedural-SQL import with explicit loss diagnostics (related: #16).
Governed parameter binding and source-side query/function pushdown.
As-of/top-one lookup operators and richer window functions.
Multi-output success/warning/error branches and human correction loops (related: #7 and Workflow).
Transactional staging/upsert effects and idempotent publication.
Fixed-width, delimited, Excel, and report output profiles (related: #8 and Reporting).

Acceptance

The synthetic contracts execute through the normal validator and reference runtime, every new node is registry-backed (validation, schema propagation, execution, and SQL support or explicit loss diagnostics), and the documentation states exactly which portions of each production flow are directly modelable, decomposable, or still unsupported.

## Context Two production SQL flows were reviewed as concrete Dataflow acceptance cases: - `Products/govoplan/heico.sql`: an Oracle CTE pipeline for student status selection and CSV output. - `Products/govoplan/rele_sql`: a PostgreSQL/PLpgSQL personnel-cost transformation and booking/report export pipeline. The supplied `rele.sql` path currently exists as `rele_sql`. HEICO is predominantly a declarative data graph. RELE combines set-based transformations with staging, validation/error streams, procedural branching, ordered mutation, grouped report generation, and formatted outputs; the latter must therefore be decomposed across Dataflow, Workflow, Datasources, Reporting, and Files rather than represented as one opaque SQL node. No production rows or third-party implementation code should be committed. Use small synthetic fixtures that preserve the semantics. ## Immediate slice - [x] Add an ordered multi-column calculation block for calculated and conditional columns. - [x] Extend the safe expression profile for the date/text/predicate functions required by the fixtures. - [x] Add semi/anti join semantics for `EXISTS`/`NOT EXISTS` flows. - [x] Add partitioned row-number/rank semantics for latest/first-per-group selection. - [x] Add executable HEICO-shaped and RELE-shaped golden fixtures with exact expected outputs. - [x] Record the coverage matrix and decomposition boundary in repository documentation. ## Follow-up scope - SQL/CTE and procedural-SQL import with explicit loss diagnostics (related: #16). - Governed parameter binding and source-side query/function pushdown. - As-of/top-one lookup operators and richer window functions. - Multi-output success/warning/error branches and human correction loops (related: #7 and Workflow). - Transactional staging/upsert effects and idempotent publication. - Fixed-width, delimited, Excel, and report output profiles (related: #8 and Reporting). ## Acceptance The synthetic contracts execute through the normal validator and reference runtime, every new node is registry-backed (validation, schema propagation, execution, and SQL support or explicit loss diagnostics), and the documentation states exactly which portions of each production flow are directly modelable, decomposable, or still unsupported.

zemion added the module/dataflow module/datasources area/api module/reporting area/webui codex/ready module/workflow

labels 2026-07-30 17:58:49 +02:00

zemion referenced this issue from GovOPlaN/govoplan-reporting

2026-07-30 17:58:53 +02:00

[Task] Assess and plan SuperX BI capability parity #3

zemion commented

2026-07-30 18:19:55 +02:00

Codex State: progress

Summary

Assessed the supplied HEICO Oracle query and RELE PL/pgSQL program against the module boundary.
Implemented ordered multi-column calculations, semi/anti joins, partitioned row_number/rank/dense_rank, and the safe predicate/string/date expression vocabulary required by the representative flows.
Added synthetic executable HEICO and RELE golden contracts; no production rows were copied.

Changed Files

README.md
docs/HEICO_RELE_ASSESSMENT.md
src/govoplan_dataflow/backend
webui/src/features/dataflow
fixtures/golden/heico-student-status
fixtures/golden/rele-booking-transform

Verification

python -m unittest discover -s tests: 67 passed
npm run typecheck: passed
npm run test:structure: passed
ruff check src tests: passed

Next / Blocked

Keep the issue open for multi-CTE/window SQL import, as-of lookup, multi-output/error branches, governed effects, and formatted publication targets documented in the assessment.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Assessed the supplied HEICO Oracle query and RELE PL/pgSQL program against the module boundary. - Implemented ordered multi-column calculations, semi/anti joins, partitioned row_number/rank/dense_rank, and the safe predicate/string/date expression vocabulary required by the representative flows. - Added synthetic executable HEICO and RELE golden contracts; no production rows were copied. ### Changed Files - `README.md` - `docs/HEICO_RELE_ASSESSMENT.md` - `src/govoplan_dataflow/backend` - `webui/src/features/dataflow` - `fixtures/golden/heico-student-status` - `fixtures/golden/rele-booking-transform` ### Verification - `python -m unittest discover -s tests: 67 passed` - `npm run typecheck: passed` - `npm run test:structure: passed` - `ruff check src tests: passed` ### Next / Blocked - Keep the issue open for multi-CTE/window SQL import, as-of lookup, multi-output/error branches, governed effects, and formatted publication targets documented in the assessment. Suggested status label: `status/in-progress`

zemion commented

2026-07-30 23:21:27 +02:00

Codex State: done

Summary

Verified that the complete immediate scope and acceptance criteria are implemented and already pushed in commit e091042.
Ordered calculations, expanded safe expressions, semi/anti joins, partitioned ranking, synthetic HEICO/RELE golden contracts, and the coverage/decomposition matrix are all registry-backed.
The broader import, reconciliation, and governed publication work remains independently tracked by Dataflow #16, #7, and #8; it is not a blocker for this acceptance issue.

Verification

python -m unittest discover -s tests: 67 passed
python -m ruff check src tests: passed
npm run typecheck: passed
npm run test:structure: passed

## Codex State: done ### Summary - Verified that the complete immediate scope and acceptance criteria are implemented and already pushed in commit e091042. - Ordered calculations, expanded safe expressions, semi/anti joins, partitioned ranking, synthetic HEICO/RELE golden contracts, and the coverage/decomposition matrix are all registry-backed. - The broader import, reconciliation, and governed publication work remains independently tracked by Dataflow #16, #7, and #8; it is not a blocker for this acceptance issue. ### Verification - `python -m unittest discover -s tests: 67 passed` - `python -m ruff check src tests: passed` - `npm run typecheck: passed` - `npm run test:structure: passed`

zemion closed this issue

2026-07-30 23:21:27 +02:00

zemion removed the

status

in-progress

codex/ready labels 2026-07-30 23:21:56 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-dataflow#17