[Feature] Execute bounded previews with node diagnostics and lineage #4

New Issue

zemion · 2026-07-28T00:58:08+02:00

zemion commented

2026-07-28 00:58:08 +02:00

Parent: #1

Scope

Execute validated pipelines against bounded tabular inputs for interactive preview. Return output schema/rows and diagnostics for each node while recording reproducibility metadata.

Acceptance Criteria

Preview has explicit row, byte, time, and node limits.
Filter, project, aggregate, sort, and limit nodes have deterministic behavior.
Every node reports input/output row counts, inferred schema, warnings, and errors.
Preview lineage records pipeline revision, executor version, source fingerprints, and result summary.
Failures do not partially publish output or leak source values into logs.
Tests cover invalid values, empty sets, nulls, limits, and deterministic reruns.

Parent: https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/1 ## Scope Execute validated pipelines against bounded tabular inputs for interactive preview. Return output schema/rows and diagnostics for each node while recording reproducibility metadata. ## Acceptance Criteria - [x] Preview has explicit row, byte, time, and node limits. - [x] Filter, project, aggregate, sort, and limit nodes have deterministic behavior. - [x] Every node reports input/output row counts, inferred schema, warnings, and errors. - [x] Preview lineage records pipeline revision, executor version, source fingerprints, and result summary. - [x] Failures do not partially publish output or leak source values into logs. - [x] Tests cover invalid values, empty sets, nulls, limits, and deterministic reruns.

zemion added the

labels 2026-07-28 00:58:08 +02:00

zemion referenced this issue

2026-07-28 01:32:40 +02:00

[Epic] Deliver a governed Dataflow manager with graphical and SQL editing #1

zemion commented

2026-07-28 01:32:41 +02:00

Codex State: progress

Summary

Implemented deterministic bounded inline execution with row, source-size, node-count, result-byte, and wall-time limits.
Successful saved previews record revision, executor version, source fingerprint, result schema/counts, diagnostics, and actor without persisting returned rows.

Changed Files

src/govoplan_dataflow/backend/executor.py
src/govoplan_dataflow/backend/service.py
tests/test_graph_and_sql.py
tests/test_service.py

Verification

Preview tests cover truncation, grouping, empty input, null ordering, invalid runtime comparisons, deterministic output, and persisted lineage.

Next / Blocked

Preserve completed-node diagnostics on failed runs and add direct resource-limit regression tests before closing this package.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Implemented deterministic bounded inline execution with row, source-size, node-count, result-byte, and wall-time limits. - Successful saved previews record revision, executor version, source fingerprint, result schema/counts, diagnostics, and actor without persisting returned rows. ### Changed Files - `src/govoplan_dataflow/backend/executor.py` - `src/govoplan_dataflow/backend/service.py` - `tests/test_graph_and_sql.py` - `tests/test_service.py` ### Verification - `Preview tests cover truncation, grouping, empty input, null ordering, invalid runtime comparisons, deterministic output, and persisted lineage.` ### Next / Blocked - Preserve completed-node diagnostics on failed runs and add direct resource-limit regression tests before closing this package. Suggested status label: `status/in-progress`

zemion closed this issue

2026-07-28 11:18:40 +02:00

zemion commented

2026-07-28 11:18:40 +02:00

Codex State: done

Summary

Implemented deterministic inline and connector-backed previews for filter, select, aggregate, sort, limit, union, join, distinct, derive, and output nodes.
Limits are explicit: 100 graph nodes, 10 sources, 250 preview rows per source, 10,000 intermediate rows, 1 MiB per node result, two seconds, and at most 500 returned rows.
Successful and failed runs preserve per-node diagnostics, executor/definition versions, source fingerprints, input/output counts, and failure lineage without persisting preview rows.
All acceptance criteria are complete in dee8380; closing the issue.

Verification

python -m unittest discover -s govoplan-connectors/tests (7 passed)
python -m unittest discover -s govoplan-dataflow/tests (24 passed)
tools/checks/check-contracts.sh (45 contracts passed)
tools/checks/check-focused.sh (core, module, API, and WebUI checks passed)
Composed JSON/CSV snapshot and connector-backed preview probe passed.

## Codex State: done ### Summary - Implemented deterministic inline and connector-backed previews for filter, select, aggregate, sort, limit, union, join, distinct, derive, and output nodes. - Limits are explicit: 100 graph nodes, 10 sources, 250 preview rows per source, 10,000 intermediate rows, 1 MiB per node result, two seconds, and at most 500 returned rows. - Successful and failed runs preserve per-node diagnostics, executor/definition versions, source fingerprints, input/output counts, and failure lineage without persisting preview rows. - All acceptance criteria are complete in `dee8380`; closing the issue. ### Verification - `python -m unittest discover -s govoplan-connectors/tests` (7 passed) - `python -m unittest discover -s govoplan-dataflow/tests` (24 passed) - `tools/checks/check-contracts.sh` (45 contracts passed) - `tools/checks/check-focused.sh` (core, module, API, and WebUI checks passed) - Composed JSON/CSV snapshot and connector-backed preview probe passed.

zemion removed the

status

in-progress

label 2026-07-28 11:19:27 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-dataflow#4