[Debt] Decompose Dataflow validation, execution, and SQL compilation paths #14

New Issue

2026-07-29T13:25:58+02:00

zemion commented

2026-07-29 13:25:58 +02:00

Problem

Dataflow has the highest remaining complexity concentration. Validation, preview execution, aggregation/derivation, run creation, SQL rendering, and SQL compilation currently combine too many node kinds and policy decisions in single functions. This raises correctness and resource-governance risk as the node library grows.

Hotspots

execute_preview, _aggregate_rows, _derive_value, _validate_node_config, _validate_graph_schemas, validate_graph, start_pipeline_run, render_sql, and compile_sql.

Acceptance criteria

Node kinds use a registry of typed validators/executors/SQL compilers instead of central conditional growth.
Graph-level validation remains separate from node-local validation.
Preview/run limits, authorization, and datasource provenance are enforced once at clear boundaries.
SQL round-trip and graph-position stability tests remain green.
Representative large-graph tests establish CPU/query/memory bounds.
Dataflow passes the configured Xenon threshold.

## Problem Dataflow has the highest remaining complexity concentration. Validation, preview execution, aggregation/derivation, run creation, SQL rendering, and SQL compilation currently combine too many node kinds and policy decisions in single functions. This raises correctness and resource-governance risk as the node library grows. ## Hotspots `execute_preview`, `_aggregate_rows`, `_derive_value`, `_validate_node_config`, `_validate_graph_schemas`, `validate_graph`, `start_pipeline_run`, `render_sql`, and `compile_sql`. ## Acceptance criteria - Node kinds use a registry of typed validators/executors/SQL compilers instead of central conditional growth. - Graph-level validation remains separate from node-local validation. - Preview/run limits, authorization, and datasource provenance are enforced once at clear boundaries. - SQL round-trip and graph-position stability tests remain green. - Representative large-graph tests establish CPU/query/memory bounds. - Dataflow passes the configured Xenon threshold.

zemion added the module/dataflow

type

debt

area/api area/db area/security source/security-audit codex/ready

labels 2026-07-29 13:25:58 +02:00

zemion commented

2026-07-29 14:24:40 +02:00

Codex State: reviewed

Reviewed after published Dataflow 09c9808. This remains open: fresh Xenon still reports F-ranked validation/execution/SQL compilation paths, including preview execution and graph/SQL validators. The pure planning/decision layers and configured threshold are not complete.

## Codex State: reviewed Reviewed after published Dataflow `09c9808`. This remains open: fresh Xenon still reports F-ranked validation/execution/SQL compilation paths, including preview execution and graph/SQL validators. The pure planning/decision layers and configured threshold are not complete.

zemion commented

2026-07-29 15:52:25 +02:00

Codex State: progress

Summary

Pushed a typed operator registry covering validator, executor, schema-propagator, and SQL-renderer facets; preview execution and SQL transform rendering now dispatch through registered operator callbacks.
Added registry coverage tests so every announced node must expose validator, executor, and explicit SQL support or rejection.

Verification

47 Dataflow tests and Ruff passed
Xenon was rerun with the audit thresholds

Next / Blocked

Do not close yet: legacy node-local validation/schema propagation and top-level planning functions still exceed Xenon C/B/A thresholds. The current run reports execute_preview E, expression evaluation F, node validation/schema propagation F, compile/render SQL F, and module/average complexity failures.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Pushed a typed operator registry covering validator, executor, schema-propagator, and SQL-renderer facets; preview execution and SQL transform rendering now dispatch through registered operator callbacks. - Added registry coverage tests so every announced node must expose validator, executor, and explicit SQL support or rejection. ### Verification - `47 Dataflow tests and Ruff passed` - `Xenon was rerun with the audit thresholds` ### Next / Blocked - Do not close yet: legacy node-local validation/schema propagation and top-level planning functions still exceed Xenon C/B/A thresholds. The current run reports execute_preview E, expression evaluation F, node validation/schema propagation F, compile/render SQL F, and module/average complexity failures. Suggested status label: `status/in-progress`

zemion added

and removed

labels 2026-07-29 15:54:02 +02:00

zemion commented

2026-07-29 16:27:51 +02:00

Codex State: done

Summary

Replaced central validation, schema propagation, execution, and SQL dispatch with the operator runtime registry.
Split graph, preview, SQL, expression, and run lifecycle hotspots; Xenon now passes absolute C/module B/average A.
Added maximum-size graph execution coverage and complete validator/schema/executor/SQL facet coverage checks.

Changed Files

src/govoplan_dataflow/backend/graph.py
src/govoplan_dataflow/backend/schema_validation.py
src/govoplan_dataflow/backend/executor.py
src/govoplan_dataflow/backend/expressions.py
src/govoplan_dataflow/backend/sql_compiler.py
src/govoplan_dataflow/backend/service.py
tests/test_operators.py

Verification

python -m unittest discover -s tests: 48 passed
ruff check src tests: passed
xenon --max-absolute C --max-modules B --max-average A src: passed

Next / Blocked

Typed execution backend work continues in #15 on top of this registry boundary.

## Codex State: done ### Summary - Replaced central validation, schema propagation, execution, and SQL dispatch with the operator runtime registry. - Split graph, preview, SQL, expression, and run lifecycle hotspots; Xenon now passes absolute C/module B/average A. - Added maximum-size graph execution coverage and complete validator/schema/executor/SQL facet coverage checks. ### Changed Files - `src/govoplan_dataflow/backend/graph.py` - `src/govoplan_dataflow/backend/schema_validation.py` - `src/govoplan_dataflow/backend/executor.py` - `src/govoplan_dataflow/backend/expressions.py` - `src/govoplan_dataflow/backend/sql_compiler.py` - `src/govoplan_dataflow/backend/service.py` - `tests/test_operators.py` ### Verification - `python -m unittest discover -s tests: 48 passed` - `ruff check src tests: passed` - `xenon --max-absolute C --max-modules B --max-average A src: passed` ### Next / Blocked - Typed execution backend work continues in #15 on top of this registry boundary.

zemion closed this issue

2026-07-29 16:27:51 +02:00

zemion referenced this issue from GovOPlaN/govoplan

2026-09-08 05:27:54 +02:00

[Roadmap] Deliver governed dataflows, search, and established-tool integration #19

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-dataflow#14