Sync wiki from project files

2026-07-28 13:52:16 +02:00
parent caa8974b11
commit 1ff2b5249f

@@ -1,4 +1,4 @@
<!-- codex-wiki-sync:1111d9caae48b86897d97060 -->
<!-- codex-wiki-sync:77773a7ff5a08dd2cea6eccb -->
> Mirrored from `/mnt/DATA/git/govoplan/docs/DATASOURCE_AND_DEFINITION_GRAPH_ARCHITECTURE.md`.
> Origin: `repository`.
@@ -30,6 +30,8 @@ The initial provider path is:
4. Any datasource may expose a frozen state for reproducible execution evidence.
5. Dataflow stores an opaque datasource reference, state policy, and expected
fingerprint.
6. A pinned Dataflow run may publish a complete bounded result as a new
immutable materialization through an idempotent Datasources capability.
Database, REST/HTTP, LDAP/directory, managed file, watched-directory, feed, and
stream providers fit behind the same origin contract. Provider-specific
@@ -58,21 +60,30 @@ kind of Dataflow or leaking either module into Core.
## Current Implementation
- Core graph and datasource contracts are versioned at `0.1.0`.
- Workflow exposes node-library discovery and definition validation APIs.
- Workflow exposes a reusable graph editor, node-library discovery, validation,
tenant-isolated definitions, immutable revisions, and activation pinning.
- Datasources exposes catalogue, origins, staging, promotion, preview,
materialization history, refresh, freeze, and retirement APIs.
materialization history, refresh, freeze, retirement, and producer
publication APIs.
- Datasources WebUI exposes all current lifecycle views.
- Connectors adapts existing tabular snapshots to datasource origins.
- Dataflow consumes only Datasources catalogue/lifecycle capabilities and can
request current, live, or latest-frozen state.
- Dataflow exposes a pinned run-lifecycle capability and a Run/Publish surface.
Its first synchronous runner records lineage and terminal state, publishes
only complete bounded results, and retains output datasource/materialization
references.
- The focused composition check proves Connector origin -> Datasource ->
pinned Dataflow run -> frozen published materialization, including replay.
## Next Slices
1. Persist and version Workflow definitions, then reuse the graph UI shell.
1. Add persisted Workflow instances, resumable transitions, human activities,
retry policy, and event subscriptions against pinned definition revisions.
2. Add SQL database and governed REST origin providers with credential-envelope
references and bounded pushdown.
3. Add managed-file and directory origins.
4. Add datasource quality rules, schema compatibility policy, retention, and
promotion approvals.
5. Publish Dataflow outputs as new materializations and expose run lifecycle to
Workflow.
5. Add asynchronous Dataflow workers and durable artifact-backed outputs for
runs that exceed the synchronous row/time/byte limits.