[Feature] Compile constrained SQL into the canonical pipeline graph #3

New Issue

zemion · 2026-07-28T00:58:08+02:00

zemion commented

2026-07-28 00:58:08 +02:00

Parent: #1

Scope

Provide a deliberately constrained, documented SQL-like dialect for tabular transformation. Parse into an AST, reject unsupported or effectful statements, compile supported queries to canonical graph nodes, and generate normalized SQL for graph pipelines that fit the dialect.

Initial Dialect

SELECT, aliases, WHERE, projection, GROUP BY, COUNT/SUM/AVG/MIN/MAX, ORDER BY, and LIMIT over registered logical sources. No DDL, DML, file access, extension loading, arbitrary functions, or pass-through execution.

Acceptance Criteria

Parsing uses a maintained SQL parser rather than regular-expression execution.
Unsupported constructs produce location-aware diagnostics.
SQL-to-graph conversion is deterministic and covered by fixtures.
Graph-to-SQL conversion clearly reports nodes that cannot be represented.
User-entered SQL is never sent unchecked to a backing database.

Parent: https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/1 ## Scope Provide a deliberately constrained, documented SQL-like dialect for tabular transformation. Parse into an AST, reject unsupported or effectful statements, compile supported queries to canonical graph nodes, and generate normalized SQL for graph pipelines that fit the dialect. ## Initial Dialect `SELECT`, aliases, `WHERE`, projection, `GROUP BY`, `COUNT/SUM/AVG/MIN/MAX`, `ORDER BY`, and `LIMIT` over registered logical sources. No DDL, DML, file access, extension loading, arbitrary functions, or pass-through execution. ## Acceptance Criteria - [x] Parsing uses a maintained SQL parser rather than regular-expression execution. - [ ] Unsupported constructs produce location-aware diagnostics. - [x] SQL-to-graph conversion is deterministic and covered by fixtures. - [x] Graph-to-SQL conversion clearly reports nodes that cannot be represented. - [x] User-entered SQL is never sent unchecked to a backing database.

zemion added the

labels 2026-07-28 00:58:08 +02:00

zemion commented

2026-07-28 01:32:16 +02:00

Codex State: progress

Summary

Implemented the first SQLGlot-backed dialect and deterministic SQL-to-graph/graph-to-SQL conversion for SELECT, WHERE, projection, grouping, aggregates, sorting, and limits.
DDL, DML, multi-statement input, joins, subqueries, unsafe functions, and unsupported clauses are rejected before execution.

Changed Files

src/govoplan_dataflow/backend/sql_compiler.py
tests/test_graph_and_sql.py

Verification

Effectful statement, multi-statement, JOIN, grouped-query roundtrip, and execution tests pass.

Next / Blocked

Add richer source-span diagnostics for every semantic rejection and fixture coverage before closing this package.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Implemented the first SQLGlot-backed dialect and deterministic SQL-to-graph/graph-to-SQL conversion for SELECT, WHERE, projection, grouping, aggregates, sorting, and limits. - DDL, DML, multi-statement input, joins, subqueries, unsafe functions, and unsupported clauses are rejected before execution. ### Changed Files - `src/govoplan_dataflow/backend/sql_compiler.py` - `tests/test_graph_and_sql.py` ### Verification - `Effectful statement, multi-statement, JOIN, grouped-query roundtrip, and execution tests pass.` ### Next / Blocked - Add richer source-span diagnostics for every semantic rejection and fixture coverage before closing this package. Suggested status label: `status/in-progress`

zemion referenced this issue

2026-07-28 01:32:40 +02:00

[Epic] Deliver a governed Dataflow manager with graphical and SQL editing #1

zemion referenced this issue

2026-07-28 11:18:40 +02:00

[Epic] Deliver a governed Dataflow manager with graphical and SQL editing #1

zemion commented

2026-07-28 11:18:40 +02:00

Codex State: progress

Summary

sqlglot AST compilation now supports the initial dialect plus deterministic DISTINCT and one explicit two-source equi-join.
Effectful statements, multiple statements, subqueries, arbitrary expressions/functions, and unrepresentable graph nodes are rejected without pass-through execution.
Graph-to-SQL-to-graph fixtures cover supported joins and transformations.

Verification

python -m unittest discover -s govoplan-connectors/tests (7 passed)
python -m unittest discover -s govoplan-dataflow/tests (24 passed)
tools/checks/check-contracts.sh (45 contracts passed)
tools/checks/check-focused.sh (core, module, API, and WebUI checks passed)
Composed JSON/CSV snapshot and connector-backed preview probe passed.

Next / Blocked

Keep this issue open until unsupported clauses and expressions carry source line/column ranges; parse errors already report line and column.

## Codex State: progress ### Summary - `sqlglot` AST compilation now supports the initial dialect plus deterministic `DISTINCT` and one explicit two-source equi-join. - Effectful statements, multiple statements, subqueries, arbitrary expressions/functions, and unrepresentable graph nodes are rejected without pass-through execution. - Graph-to-SQL-to-graph fixtures cover supported joins and transformations. ### Verification - `python -m unittest discover -s govoplan-connectors/tests` (7 passed) - `python -m unittest discover -s govoplan-dataflow/tests` (24 passed) - `tools/checks/check-contracts.sh` (45 contracts passed) - `tools/checks/check-focused.sh` (core, module, API, and WebUI checks passed) - Composed JSON/CSV snapshot and connector-backed preview probe passed. ### Next / Blocked - Keep this issue open until unsupported clauses and expressions carry source line/column ranges; parse errors already report line and column.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-dataflow#3