GovOPlaN/govoplan
1
0
Fork 0
Code Issues 10 Pull Requests Actions 3 Packages Projects Releases Wiki Activity
Files
6afb8fea76bdce5cbae133610b01abb6200d1af8
govoplan/docs/DATASOURCE_AND_DEFINITION_GRAPH_ARCHITECTURE.md
Albrecht Degering 163b35c0af
Some checks failed
Dependency Audit / dependency-audit (push) Has been cancelled
Details
Security Audit / security-audit (push) Has been cancelled
Details
feat: integrate datasource and workflow modules
2026-07-28 12:45:05 +02:00

3.2 KiB
Raw Blame History

Datasource And Definition Graph Architecture

Two-Layer Data Boundary

GovOPlaN separates governed data identity from external acquisition:

Layer Owner Responsibilities
Datasource layer govoplan-datasources Catalogue, tenant visibility, live/cached/static mode, staging, immutable materializations, frozen states, schema, fingerprints, provenance, and bounded reads
Connector layer govoplan-connectors and protocol/provider modules External protocols, endpoints, connection profiles, credentials, discovery, provider health, source-side filtering, and query pushdown

Connectors publish versioned datasource origins. Datasources registers those origins and presents one stable capability to Dataflow, Workflow, Reporting, Risk Compliance, and other consumers. Consumers must not import connector implementations or retain credentials.

The initial provider path is:

  1. Connectors imports a bounded JSON/CSV snapshot and exposes it as an origin.
  2. Datasources registers it as live or cached, or accepts a direct static upload through staging.
  3. Cached refreshes and static promotions append immutable materializations.
  4. Any datasource may expose a frozen state for reproducible execution evidence.
  5. Dataflow stores an opaque datasource reference, state policy, and expected fingerprint.

Database, REST/HTTP, LDAP/directory, managed file, watched-directory, feed, and stream providers fit behind the same origin contract. Provider-specific configuration remains in Connectors.

Shared Definition Graph

Core owns domain-neutral graph primitives:

  • nodes, typed ports, edges, and configuration field descriptors;
  • node libraries and category labels;
  • graph size, connectivity, cycle, and node-count constraints;
  • shared backend validation and frontend connection checks.

Domain modules own their semantics:

  • Dataflow provides load, combine, filter, transform, and output nodes. Its graph is acyclic and has one output.
  • Workflow provides trigger, activity, review, decision, wait, module-action, Dataflow, and outcome nodes. It permits governed loops and has exactly one trigger plus one or more outcomes.

This division permits a shared editor shell without making Workflow a special kind of Dataflow or leaking either module into Core.

Current Implementation

  • Core graph and datasource contracts are versioned at 0.1.0.
  • Workflow exposes node-library discovery and definition validation APIs.
  • Datasources exposes catalogue, origins, staging, promotion, preview, materialization history, refresh, freeze, and retirement APIs.
  • Datasources WebUI exposes all current lifecycle views.
  • Connectors adapts existing tabular snapshots to datasource origins.
  • Dataflow consumes only Datasources catalogue/lifecycle capabilities and can request current, live, or latest-frozen state.

Next Slices

  1. Persist and version Workflow definitions, then reuse the graph UI shell.
  2. Add SQL database and governed REST origin providers with credential-envelope references and bounded pushdown.
  3. Add managed-file and directory origins.
  4. Add datasource quality rules, schema compatibility policy, retention, and promotion approvals.
  5. Publish Dataflow outputs as new materializations and expose run lifecycle to Workflow.