feat: add extensible operators and golden flows

This commit is contained in:
2026-07-29 15:50:15 +02:00
parent 09c98087c5
commit 946202ef01
27 changed files with 3584 additions and 218 deletions
+22 -2
View File
@@ -28,8 +28,9 @@ nodes by purpose:
| --- | --- |
| Load | Inline data, datasource |
| Combine | Append rows, join tables |
| Filter | Filter rows, remove duplicates |
| Transform | Select columns, derive column, aggregate, sort, limit |
| Filter | Column filter, typed-expression filter, remove duplicates |
| Transform | Select, derive, typed expression, conversion, replacement, aggregate, sort, limit, reusable subflow |
| Quality | Quality rules, keyed reconciliation |
| Output | Preview output |
Join nodes have explicit left and right ports. Append nodes accept two or more
@@ -53,6 +54,15 @@ statement into the canonical graph. The dialect supports projection, aliases,
filters, grouping, aggregate functions, sorting, limits, `DISTINCT`, append,
and one two-source equi-join. It rejects DDL, DML, arbitrary subqueries,
arbitrary functions, file access, and unchecked pass-through execution.
The typed-expression library is a separate allowlisted AST evaluator shared by
expression filters and calculated columns. It supports literals, columns,
arithmetic, comparisons, boolean logic, `CASE`, safe casts, and a bounded
string/numeric function catalogue without Python evaluation or effectful SQL.
Node definitions, validators, preview executors, and SQL renderers are
registered independently in the operator registry. Adding a node no longer
requires another branch in the preview or graph-to-SQL dispatch loop. Operators
that cannot be represented by constrained SQL declare that explicitly.
Preview reads at most 250 rows per source and enforces time, intermediate-row,
result-byte, graph-node, and response-row bounds. Saved previews record the
@@ -89,6 +99,16 @@ permissions block the delivery before source access or output publication.
Confidential and restricted events are not accepted through the direct
ingress; those require Core's transactional event bridge.
Reusable subflow nodes pin a template reference, version, graph snapshot, and
parameter values. Their single input is bound to an explicitly marked inline
source inside the snapshot, parameter substitution is data-only, and nesting
is bounded. This keeps completed run definitions reproducible even when the
source template changes later.
The executable fixtures in `fixtures/golden` cover the monthly structured-file
reconciliation story and the sanctions-screening story with reviewable sample
inputs and exact expected outputs.
## Development
```bash