91 lines
4.8 KiB
Markdown
91 lines
4.8 KiB
Markdown
# Datasource And Definition Graph Architecture
|
|
|
|
## Two-Layer Data Boundary
|
|
|
|
GovOPlaN separates governed data identity from external acquisition:
|
|
|
|
| Layer | Owner | Responsibilities |
|
|
| --- | --- | --- |
|
|
| Datasource layer | `govoplan-datasources` | Governed data/register catalogue, tenant visibility, live/cached/static mode, source authority, staging, immutable materializations, frozen states, schema, quality/freshness policy, institutional provenance, dependencies, and bounded reads |
|
|
| Connector layer | `govoplan-connectors` and protocol/provider modules | External protocols, endpoints, connection profiles, credentials, discovery, provider maturity/authority support, health, source-side filtering, query pushdown, and effect reconciliation |
|
|
|
|
Connectors publish versioned datasource origins. Datasources registers those
|
|
origins and presents one stable capability to Dataflow, Workflow, Reporting,
|
|
Risk Compliance, and other consumers. Consumers must not import connector
|
|
implementations or retain credentials.
|
|
|
|
The initial provider path is:
|
|
|
|
1. Connectors imports a bounded JSON/CSV snapshot and exposes it as an origin.
|
|
2. Datasources registers it as live or cached, or accepts a direct static upload
|
|
through staging.
|
|
3. Cached refreshes and static promotions append immutable materializations.
|
|
4. Any datasource may expose a frozen state for reproducible execution evidence.
|
|
5. Dataflow stores an opaque datasource reference, state policy, and expected
|
|
fingerprint.
|
|
6. A pinned Dataflow run may publish a complete bounded result as a new
|
|
immutable materialization through an idempotent Datasources capability.
|
|
|
|
Database, REST/HTTP, LDAP/directory, managed file, watched-directory, feed, and
|
|
stream providers fit behind the same origin contract. Provider-specific
|
|
configuration remains in Connectors.
|
|
|
|
## Shared Definition Graph
|
|
|
|
Core owns domain-neutral graph primitives:
|
|
|
|
- nodes, typed ports, edges, and configuration field descriptors;
|
|
- node libraries and category labels;
|
|
- graph size, connectivity, cycle, and node-count constraints;
|
|
- shared backend validation and frontend connection checks.
|
|
|
|
Domain modules own their semantics:
|
|
|
|
- Dataflow provides load, combine, filter, transform, and output nodes. Its
|
|
graph is acyclic and has one output.
|
|
- Workflow Engine provides trigger, activity, review, decision, wait,
|
|
module-action, Dataflow, and outcome semantics. It permits governed loops and
|
|
has exactly one trigger plus one or more outcomes. The optional Workflow
|
|
module supplies the editor over the same native graph/BPMN language.
|
|
|
|
This division permits a shared editor shell without making Workflow a special
|
|
kind of Dataflow or leaking either module into Core.
|
|
|
|
## Current Implementation
|
|
|
|
- Core graph and datasource contracts are versioned at `0.1.0`.
|
|
- Workflow Engine owns tenant-isolated definitions, immutable revisions,
|
|
activation pinning, module-contributed versioned baselines, runtime instances,
|
|
governed action/effect execution, retries, waits, and reconciliation. The
|
|
optional Workflow module exposes the reusable native BPMN graph editor.
|
|
- Datasources exposes catalogue, origins, staging, promotion, preview,
|
|
materialization history, refresh, freeze, retirement, and producer
|
|
publication APIs.
|
|
- Datasources WebUI exposes all current lifecycle views.
|
|
- Connectors adapts existing tabular snapshots to datasource origins.
|
|
- Dataflow consumes only Datasources catalogue/lifecycle capabilities and can
|
|
request current, live, or latest-frozen state.
|
|
- Dataflow exposes typed graph/IR, registry-driven validation/execution/SQL
|
|
compilation, expressions and reusable subflows, a pinned run-lifecycle
|
|
capability, production worker boundary, and Run/Publish surface. Runs record
|
|
lineage and intermediate artifacts and publish only complete bounded results.
|
|
- The focused composition check proves Connector origin -> Datasource ->
|
|
pinned Dataflow run -> frozen published materialization, including replay.
|
|
|
|
## Next Slices
|
|
|
|
1. Add the provider declaration and source-authority binding used consistently
|
|
by Connectors, Datasources, configuration packages, Ops, and Docs.
|
|
2. Add typed datasource owner/steward, legal/purpose, quality/freshness,
|
|
classification, correction, service/process, and downstream dependency
|
|
metadata under
|
|
[Datasources #6](https://git.add-ideas.de/GovOPlaN/govoplan-datasources/issues/6).
|
|
3. Add SQL database and governed REST origin providers with credential-envelope
|
|
references and bounded pushdown.
|
|
4. Add managed-file and directory origins.
|
|
5. Complete datasource quality rules, schema compatibility policy, retention, and
|
|
promotion approvals.
|
|
6. Complete scheduled/event/API/chained Dataflow trigger governance, reusable
|
|
template inheritance, Reporting publication, human reconciliation transforms,
|
|
and target resource/recovery evidence for large runs.
|