From 1ff2b5249f1e4497ed0d596281943d867e9337ff Mon Sep 17 00:00:00 2001 From: Albrecht Degering Date: Tue, 28 Jul 2026 13:52:16 +0200 Subject: [PATCH] Sync wiki from project files --- ...OURCE-AND-DEFINITION-GRAPH-ARCHITECTURE.md | 23 ++++++++++++++----- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/Repo-docs-DATASOURCE-AND-DEFINITION-GRAPH-ARCHITECTURE.md b/Repo-docs-DATASOURCE-AND-DEFINITION-GRAPH-ARCHITECTURE.md index 2065571..432b488 100644 --- a/Repo-docs-DATASOURCE-AND-DEFINITION-GRAPH-ARCHITECTURE.md +++ b/Repo-docs-DATASOURCE-AND-DEFINITION-GRAPH-ARCHITECTURE.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan/docs/DATASOURCE_AND_DEFINITION_GRAPH_ARCHITECTURE.md`. > Origin: `repository`. @@ -30,6 +30,8 @@ The initial provider path is: 4. Any datasource may expose a frozen state for reproducible execution evidence. 5. Dataflow stores an opaque datasource reference, state policy, and expected fingerprint. +6. A pinned Dataflow run may publish a complete bounded result as a new + immutable materialization through an idempotent Datasources capability. Database, REST/HTTP, LDAP/directory, managed file, watched-directory, feed, and stream providers fit behind the same origin contract. Provider-specific @@ -58,21 +60,30 @@ kind of Dataflow or leaking either module into Core. ## Current Implementation - Core graph and datasource contracts are versioned at `0.1.0`. -- Workflow exposes node-library discovery and definition validation APIs. +- Workflow exposes a reusable graph editor, node-library discovery, validation, + tenant-isolated definitions, immutable revisions, and activation pinning. - Datasources exposes catalogue, origins, staging, promotion, preview, - materialization history, refresh, freeze, and retirement APIs. + materialization history, refresh, freeze, retirement, and producer + publication APIs. - Datasources WebUI exposes all current lifecycle views. - Connectors adapts existing tabular snapshots to datasource origins. - Dataflow consumes only Datasources catalogue/lifecycle capabilities and can request current, live, or latest-frozen state. +- Dataflow exposes a pinned run-lifecycle capability and a Run/Publish surface. + Its first synchronous runner records lineage and terminal state, publishes + only complete bounded results, and retains output datasource/materialization + references. +- The focused composition check proves Connector origin -> Datasource -> + pinned Dataflow run -> frozen published materialization, including replay. ## Next Slices -1. Persist and version Workflow definitions, then reuse the graph UI shell. +1. Add persisted Workflow instances, resumable transitions, human activities, + retry policy, and event subscriptions against pinned definition revisions. 2. Add SQL database and governed REST origin providers with credential-envelope references and bounded pushdown. 3. Add managed-file and directory origins. 4. Add datasource quality rules, schema compatibility policy, retention, and promotion approvals. -5. Publish Dataflow outputs as new materializations and expose run lifecycle to - Workflow. +5. Add asynchronous Dataflow workers and durable artifact-backed outputs for + runs that exceed the synchronous row/time/byte limits.