83 lines
4.0 KiB
Markdown
83 lines
4.0 KiB
Markdown
# Datasource And Definition Graph Architecture
|
|
|
|
## Two-Layer Data Boundary
|
|
|
|
GovOPlaN separates governed data identity from external acquisition:
|
|
|
|
| Layer | Owner | Responsibilities |
|
|
| --- | --- | --- |
|
|
| Datasource layer | `govoplan-datasources` | Catalogue, tenant visibility, live/cached/static mode, staging, immutable materializations, frozen states, schema, fingerprints, provenance, and bounded reads |
|
|
| Connector layer | `govoplan-connectors` and protocol/provider modules | External protocols, endpoints, connection profiles, credentials, discovery, provider health, source-side filtering, and query pushdown |
|
|
|
|
Connectors publish versioned datasource origins. Datasources registers those
|
|
origins and presents one stable capability to Dataflow, Workflow, Reporting,
|
|
Risk Compliance, and other consumers. Consumers must not import connector
|
|
implementations or retain credentials.
|
|
|
|
The initial provider path is:
|
|
|
|
1. Connectors imports a bounded JSON/CSV snapshot and exposes it as an origin.
|
|
2. Datasources registers it as live or cached, or accepts a direct static upload
|
|
through staging.
|
|
3. Cached refreshes and static promotions append immutable materializations.
|
|
4. Any datasource may expose a frozen state for reproducible execution evidence.
|
|
5. Dataflow stores an opaque datasource reference, state policy, and expected
|
|
fingerprint.
|
|
6. A pinned Dataflow run may publish a complete bounded result as a new
|
|
immutable materialization through an idempotent Datasources capability.
|
|
|
|
Database, REST/HTTP, LDAP/directory, managed file, watched-directory, feed, and
|
|
stream providers fit behind the same origin contract. Provider-specific
|
|
configuration remains in Connectors.
|
|
|
|
## Shared Definition Graph
|
|
|
|
Core owns domain-neutral graph primitives:
|
|
|
|
- nodes, typed ports, edges, and configuration field descriptors;
|
|
- node libraries and category labels;
|
|
- graph size, connectivity, cycle, and node-count constraints;
|
|
- shared backend validation and frontend connection checks.
|
|
|
|
Domain modules own their semantics:
|
|
|
|
- Dataflow provides load, combine, filter, transform, and output nodes. Its
|
|
graph is acyclic and has one output.
|
|
- Workflow provides trigger, activity, review, decision, wait, module-action,
|
|
Dataflow, and outcome nodes. It permits governed loops and has exactly one
|
|
trigger plus one or more outcomes.
|
|
|
|
This division permits a shared editor shell without making Workflow a special
|
|
kind of Dataflow or leaking either module into Core.
|
|
|
|
## Current Implementation
|
|
|
|
- Core graph and datasource contracts are versioned at `0.1.0`.
|
|
- Workflow exposes a reusable graph editor, node-library discovery, validation,
|
|
tenant-isolated definitions, immutable revisions, and activation pinning.
|
|
- Datasources exposes catalogue, origins, staging, promotion, preview,
|
|
materialization history, refresh, freeze, retirement, and producer
|
|
publication APIs.
|
|
- Datasources WebUI exposes all current lifecycle views.
|
|
- Connectors adapts existing tabular snapshots to datasource origins.
|
|
- Dataflow consumes only Datasources catalogue/lifecycle capabilities and can
|
|
request current, live, or latest-frozen state.
|
|
- Dataflow exposes a pinned run-lifecycle capability and a Run/Publish surface.
|
|
Its first synchronous runner records lineage and terminal state, publishes
|
|
only complete bounded results, and retains output datasource/materialization
|
|
references.
|
|
- The focused composition check proves Connector origin -> Datasource ->
|
|
pinned Dataflow run -> frozen published materialization, including replay.
|
|
|
|
## Next Slices
|
|
|
|
1. Add persisted Workflow instances, resumable transitions, human activities,
|
|
retry policy, and event subscriptions against pinned definition revisions.
|
|
2. Add SQL database and governed REST origin providers with credential-envelope
|
|
references and bounded pushdown.
|
|
3. Add managed-file and directory origins.
|
|
4. Add datasource quality rules, schema compatibility policy, retention, and
|
|
promotion approvals.
|
|
5. Add asynchronous Dataflow workers and durable artifact-backed outputs for
|
|
runs that exceed the synchronous row/time/byte limits.
|