feat: add extensible operators and golden flows
This commit is contained in:
@@ -28,8 +28,9 @@ nodes by purpose:
|
||||
| --- | --- |
|
||||
| Load | Inline data, datasource |
|
||||
| Combine | Append rows, join tables |
|
||||
| Filter | Filter rows, remove duplicates |
|
||||
| Transform | Select columns, derive column, aggregate, sort, limit |
|
||||
| Filter | Column filter, typed-expression filter, remove duplicates |
|
||||
| Transform | Select, derive, typed expression, conversion, replacement, aggregate, sort, limit, reusable subflow |
|
||||
| Quality | Quality rules, keyed reconciliation |
|
||||
| Output | Preview output |
|
||||
|
||||
Join nodes have explicit left and right ports. Append nodes accept two or more
|
||||
@@ -53,6 +54,15 @@ statement into the canonical graph. The dialect supports projection, aliases,
|
||||
filters, grouping, aggregate functions, sorting, limits, `DISTINCT`, append,
|
||||
and one two-source equi-join. It rejects DDL, DML, arbitrary subqueries,
|
||||
arbitrary functions, file access, and unchecked pass-through execution.
|
||||
The typed-expression library is a separate allowlisted AST evaluator shared by
|
||||
expression filters and calculated columns. It supports literals, columns,
|
||||
arithmetic, comparisons, boolean logic, `CASE`, safe casts, and a bounded
|
||||
string/numeric function catalogue without Python evaluation or effectful SQL.
|
||||
|
||||
Node definitions, validators, preview executors, and SQL renderers are
|
||||
registered independently in the operator registry. Adding a node no longer
|
||||
requires another branch in the preview or graph-to-SQL dispatch loop. Operators
|
||||
that cannot be represented by constrained SQL declare that explicitly.
|
||||
|
||||
Preview reads at most 250 rows per source and enforces time, intermediate-row,
|
||||
result-byte, graph-node, and response-row bounds. Saved previews record the
|
||||
@@ -89,6 +99,16 @@ permissions block the delivery before source access or output publication.
|
||||
Confidential and restricted events are not accepted through the direct
|
||||
ingress; those require Core's transactional event bridge.
|
||||
|
||||
Reusable subflow nodes pin a template reference, version, graph snapshot, and
|
||||
parameter values. Their single input is bound to an explicitly marked inline
|
||||
source inside the snapshot, parameter substitution is data-only, and nesting
|
||||
is bounded. This keeps completed run definitions reproducible even when the
|
||||
source template changes later.
|
||||
|
||||
The executable fixtures in `fixtures/golden` cover the monthly structured-file
|
||||
reconciliation story and the sanctions-screening story with reviewable sample
|
||||
inputs and exact expected outputs.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user