Sync wiki from project files
@@ -1,4 +1,4 @@
|
||||
<!-- codex-wiki-sync:1111d9caae48b86897d97060 -->
|
||||
<!-- codex-wiki-sync:77773a7ff5a08dd2cea6eccb -->
|
||||
|
||||
> Mirrored from `/mnt/DATA/git/govoplan/docs/DATASOURCE_AND_DEFINITION_GRAPH_ARCHITECTURE.md`.
|
||||
> Origin: `repository`.
|
||||
@@ -30,6 +30,8 @@ The initial provider path is:
|
||||
4. Any datasource may expose a frozen state for reproducible execution evidence.
|
||||
5. Dataflow stores an opaque datasource reference, state policy, and expected
|
||||
fingerprint.
|
||||
6. A pinned Dataflow run may publish a complete bounded result as a new
|
||||
immutable materialization through an idempotent Datasources capability.
|
||||
|
||||
Database, REST/HTTP, LDAP/directory, managed file, watched-directory, feed, and
|
||||
stream providers fit behind the same origin contract. Provider-specific
|
||||
@@ -58,21 +60,30 @@ kind of Dataflow or leaking either module into Core.
|
||||
## Current Implementation
|
||||
|
||||
- Core graph and datasource contracts are versioned at `0.1.0`.
|
||||
- Workflow exposes node-library discovery and definition validation APIs.
|
||||
- Workflow exposes a reusable graph editor, node-library discovery, validation,
|
||||
tenant-isolated definitions, immutable revisions, and activation pinning.
|
||||
- Datasources exposes catalogue, origins, staging, promotion, preview,
|
||||
materialization history, refresh, freeze, and retirement APIs.
|
||||
materialization history, refresh, freeze, retirement, and producer
|
||||
publication APIs.
|
||||
- Datasources WebUI exposes all current lifecycle views.
|
||||
- Connectors adapts existing tabular snapshots to datasource origins.
|
||||
- Dataflow consumes only Datasources catalogue/lifecycle capabilities and can
|
||||
request current, live, or latest-frozen state.
|
||||
- Dataflow exposes a pinned run-lifecycle capability and a Run/Publish surface.
|
||||
Its first synchronous runner records lineage and terminal state, publishes
|
||||
only complete bounded results, and retains output datasource/materialization
|
||||
references.
|
||||
- The focused composition check proves Connector origin -> Datasource ->
|
||||
pinned Dataflow run -> frozen published materialization, including replay.
|
||||
|
||||
## Next Slices
|
||||
|
||||
1. Persist and version Workflow definitions, then reuse the graph UI shell.
|
||||
1. Add persisted Workflow instances, resumable transitions, human activities,
|
||||
retry policy, and event subscriptions against pinned definition revisions.
|
||||
2. Add SQL database and governed REST origin providers with credential-envelope
|
||||
references and bounded pushdown.
|
||||
3. Add managed-file and directory origins.
|
||||
4. Add datasource quality rules, schema compatibility policy, retention, and
|
||||
promotion approvals.
|
||||
5. Publish Dataflow outputs as new materializations and expose run lifecycle to
|
||||
Workflow.
|
||||
5. Add asynchronous Dataflow workers and durable artifact-backed outputs for
|
||||
runs that exceed the synchronous row/time/byte limits.
|
||||
|
||||
Reference in New Issue
Block a user