Add typed Arrow execution backend boundary

This commit is contained in:
2026-07-29 16:47:14 +02:00
parent 69509d5cc2
commit 8a11d7571b
17 changed files with 2264 additions and 49 deletions
+20 -4
View File
@@ -59,10 +59,26 @@ expression filters and calculated columns. It supports literals, columns,
arithmetic, comparisons, boolean logic, `CASE`, safe casts, and a bounded
string/numeric function catalogue without Python evaluation or effectful SQL.
Node definitions, validators, preview executors, and SQL renderers are
registered independently in the operator registry. Adding a node no longer
requires another branch in the preview or graph-to-SQL dispatch loop. Operators
that cannot be represented by constrained SQL declare that explicitly.
Node definitions, validators, schema propagators, preview executors, and SQL
renderers are registered independently in the operator registry. Adding a node
no longer requires another branch in the preview or graph-to-SQL dispatch loop.
Operators that cannot be represented by constrained SQL declare that
explicitly.
Every executable graph also has a versioned typed IR. It preserves graph
identity and layout while giving ports, schemas, expressions, parameters,
lineage, diagnostics, semantic hashes, and physical results stable contracts.
The backend boundary exchanges bounded typed columnar batches that can be
serialized as Arrow IPC. The deterministic Python executor remains the default
and fallback.
Install `.[analytics]` to enable the analytical backend. It executes only SQL
generated from validated graphs in a separate short-lived DuckDB process.
Inputs and outputs cross the process boundary as Arrow IPC; external access,
extension installation/loading, persistent secrets, configuration changes,
temporary spill files, and multiple DuckDB threads are disabled. Wall-clock,
row, byte, memory, file-descriptor, and backend-concurrency limits are applied.
Callers can explicitly request `reference`, `duckdb`, or `auto` for a preview.
Preview reads at most 250 rows per source and enforces time, intermediate-row,
result-byte, graph-node, and response-row bounds. Saved previews record the