Add typed Arrow execution backend boundary
This commit is contained in:
@@ -59,10 +59,26 @@ expression filters and calculated columns. It supports literals, columns,
|
||||
arithmetic, comparisons, boolean logic, `CASE`, safe casts, and a bounded
|
||||
string/numeric function catalogue without Python evaluation or effectful SQL.
|
||||
|
||||
Node definitions, validators, preview executors, and SQL renderers are
|
||||
registered independently in the operator registry. Adding a node no longer
|
||||
requires another branch in the preview or graph-to-SQL dispatch loop. Operators
|
||||
that cannot be represented by constrained SQL declare that explicitly.
|
||||
Node definitions, validators, schema propagators, preview executors, and SQL
|
||||
renderers are registered independently in the operator registry. Adding a node
|
||||
no longer requires another branch in the preview or graph-to-SQL dispatch loop.
|
||||
Operators that cannot be represented by constrained SQL declare that
|
||||
explicitly.
|
||||
|
||||
Every executable graph also has a versioned typed IR. It preserves graph
|
||||
identity and layout while giving ports, schemas, expressions, parameters,
|
||||
lineage, diagnostics, semantic hashes, and physical results stable contracts.
|
||||
The backend boundary exchanges bounded typed columnar batches that can be
|
||||
serialized as Arrow IPC. The deterministic Python executor remains the default
|
||||
and fallback.
|
||||
|
||||
Install `.[analytics]` to enable the analytical backend. It executes only SQL
|
||||
generated from validated graphs in a separate short-lived DuckDB process.
|
||||
Inputs and outputs cross the process boundary as Arrow IPC; external access,
|
||||
extension installation/loading, persistent secrets, configuration changes,
|
||||
temporary spill files, and multiple DuckDB threads are disabled. Wall-clock,
|
||||
row, byte, memory, file-descriptor, and backend-concurrency limits are applied.
|
||||
Callers can explicitly request `reference`, `duckdb`, or `auto` for a preview.
|
||||
|
||||
Preview reads at most 250 rows per source and enforces time, intermediate-row,
|
||||
result-byte, graph-node, and response-row bounds. Saved previews record the
|
||||
|
||||
Reference in New Issue
Block a user