[Feature] Compile constrained SQL into the canonical pipeline graph #3

New Issue

2026-07-28T00:58:08+02:00

zemion commented

2026-07-28 00:58:08 +02:00

Parent: #1

Scope

Provide a deliberately constrained, documented SQL-like dialect for tabular transformation. Parse into an AST, reject unsupported or effectful statements, compile supported queries to canonical graph nodes, and generate normalized SQL for graph pipelines that fit the dialect.

Initial Dialect

SELECT, aliases, WHERE, projection, GROUP BY, COUNT/SUM/AVG/MIN/MAX, ORDER BY, and LIMIT over registered logical sources. No DDL, DML, file access, extension loading, arbitrary functions, or pass-through execution.

Acceptance Criteria

Parsing uses a maintained SQL parser rather than regular-expression execution.
Unsupported constructs produce location-aware diagnostics.
SQL-to-graph conversion is deterministic and covered by fixtures.
Graph-to-SQL conversion clearly reports nodes that cannot be represented.
User-entered SQL is never sent unchecked to a backing database.

Parent: https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/1 ## Scope Provide a deliberately constrained, documented SQL-like dialect for tabular transformation. Parse into an AST, reject unsupported or effectful statements, compile supported queries to canonical graph nodes, and generate normalized SQL for graph pipelines that fit the dialect. ## Initial Dialect `SELECT`, aliases, `WHERE`, projection, `GROUP BY`, `COUNT/SUM/AVG/MIN/MAX`, `ORDER BY`, and `LIMIT` over registered logical sources. No DDL, DML, file access, extension loading, arbitrary functions, or pass-through execution. ## Acceptance Criteria - [x] Parsing uses a maintained SQL parser rather than regular-expression execution. - [ ] Unsupported constructs produce location-aware diagnostics. - [x] SQL-to-graph conversion is deterministic and covered by fixtures. - [x] Graph-to-SQL conversion clearly reports nodes that cannot be represented. - [x] User-entered SQL is never sent unchecked to a backing database.

zemion added the module/dataflow area/api area/security codex/ready

labels 2026-07-28 00:58:08 +02:00

zemion commented

2026-07-28 01:32:16 +02:00

Codex State: progress

Summary

Implemented the first SQLGlot-backed dialect and deterministic SQL-to-graph/graph-to-SQL conversion for SELECT, WHERE, projection, grouping, aggregates, sorting, and limits.
DDL, DML, multi-statement input, joins, subqueries, unsafe functions, and unsupported clauses are rejected before execution.

Changed Files

src/govoplan_dataflow/backend/sql_compiler.py
tests/test_graph_and_sql.py

Verification

Effectful statement, multi-statement, JOIN, grouped-query roundtrip, and execution tests pass.

Next / Blocked

Add richer source-span diagnostics for every semantic rejection and fixture coverage before closing this package.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Implemented the first SQLGlot-backed dialect and deterministic SQL-to-graph/graph-to-SQL conversion for SELECT, WHERE, projection, grouping, aggregates, sorting, and limits. - DDL, DML, multi-statement input, joins, subqueries, unsafe functions, and unsupported clauses are rejected before execution. ### Changed Files - `src/govoplan_dataflow/backend/sql_compiler.py` - `tests/test_graph_and_sql.py` ### Verification - `Effectful statement, multi-statement, JOIN, grouped-query roundtrip, and execution tests pass.` ### Next / Blocked - Add richer source-span diagnostics for every semantic rejection and fixture coverage before closing this package. Suggested status label: `status/in-progress`

zemion referenced this issue

2026-07-28 01:32:40 +02:00

[Epic] Deliver a governed Dataflow manager with graphical and SQL editing #1

zemion referenced this issue

2026-07-28 11:18:40 +02:00

[Epic] Deliver a governed Dataflow manager with graphical and SQL editing #1

zemion commented

2026-07-28 11:18:40 +02:00

Codex State: progress

Summary

sqlglot AST compilation now supports the initial dialect plus deterministic DISTINCT and one explicit two-source equi-join.
Effectful statements, multiple statements, subqueries, arbitrary expressions/functions, and unrepresentable graph nodes are rejected without pass-through execution.
Graph-to-SQL-to-graph fixtures cover supported joins and transformations.

Verification

python -m unittest discover -s govoplan-connectors/tests (7 passed)
python -m unittest discover -s govoplan-dataflow/tests (24 passed)
tools/checks/check-contracts.sh (45 contracts passed)
tools/checks/check-focused.sh (core, module, API, and WebUI checks passed)
Composed JSON/CSV snapshot and connector-backed preview probe passed.

Next / Blocked

Keep this issue open until unsupported clauses and expressions carry source line/column ranges; parse errors already report line and column.

## Codex State: progress ### Summary - `sqlglot` AST compilation now supports the initial dialect plus deterministic `DISTINCT` and one explicit two-source equi-join. - Effectful statements, multiple statements, subqueries, arbitrary expressions/functions, and unrepresentable graph nodes are rejected without pass-through execution. - Graph-to-SQL-to-graph fixtures cover supported joins and transformations. ### Verification - `python -m unittest discover -s govoplan-connectors/tests` (7 passed) - `python -m unittest discover -s govoplan-dataflow/tests` (24 passed) - `tools/checks/check-contracts.sh` (45 contracts passed) - `tools/checks/check-focused.sh` (core, module, API, and WebUI checks passed) - Composed JSON/CSV snapshot and connector-backed preview probe passed. ### Next / Blocked - Keep this issue open until unsupported clauses and expressions carry source line/column ranges; parse errors already report line and column.

zemion commented

2026-07-31 15:23:53 +02:00

Close-out audit, 2026-07-31

The SQLGlot-backed compiler, deterministic round trips, rejection of effectful/unsupported input, and no-pass-through execution are complete. The current 67-test Dataflow suite passes.

One acceptance item remains: diagnostics for unsupported syntax/constructs need a stable source span (line/column and, where available, end position) rather than only a message/code. Add normalized parser/compiler diagnostic DTOs and focused malformed/unsupported-query fixtures; this issue can then close.

## Close-out audit, 2026-07-31 The SQLGlot-backed compiler, deterministic round trips, rejection of effectful/unsupported input, and no-pass-through execution are complete. The current 67-test Dataflow suite passes. One acceptance item remains: diagnostics for unsupported syntax/constructs need a stable source span (line/column and, where available, end position) rather than only a message/code. Add normalized parser/compiler diagnostic DTOs and focused malformed/unsupported-query fixtures; this issue can then close.

zemion commented

2026-07-31 17:20:02 +02:00

Codex State: complete

Implemented normalized SQL diagnostic source locations in commit 8724591.

Parser errors now expose start/end line, column, and character offsets.
Semantic compiler rejections derive stable spans from SQLGlot AST token metadata; unsupported clauses, expressions, joins, literals, projections, grouping, ordering, and limits carry source locations.
The WebUI renders line:column for SQL diagnostics.
Focused malformed and unsupported multiline fixtures cover parser, clause, and expression diagnostics.

Verification:

python -m unittest discover -s tests — 68 passed.
Dataflow WebUI typecheck and structure test passed.
Linked full WebUI production build and bundle budget passed.

All acceptance criteria are complete.

## Codex State: complete Implemented normalized SQL diagnostic source locations in commit `8724591`. - Parser errors now expose start/end line, column, and character offsets. - Semantic compiler rejections derive stable spans from SQLGlot AST token metadata; unsupported clauses, expressions, joins, literals, projections, grouping, ordering, and limits carry source locations. - The WebUI renders `line:column` for SQL diagnostics. - Focused malformed and unsupported multiline fixtures cover parser, clause, and expression diagnostics. Verification: - `python -m unittest discover -s tests` — 68 passed. - Dataflow WebUI typecheck and structure test passed. - Linked full WebUI production build and bundle budget passed. All acceptance criteria are complete.

zemion closed this issue

2026-07-31 17:20:02 +02:00

zemion referenced this issue

2026-08-04 12:50:27 +02:00

[Feature] Publish governed datasets for Reporting and export #8

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-dataflow#3