# Datasource And Definition Graph Architecture

## Two-Layer Data Boundary

GovOPlaN separates governed data identity from external acquisition:

| Layer | Owner | Responsibilities |
| --- | --- | --- |
| Datasource layer | `govoplan-datasources` | Governed data/register catalogue, tenant visibility, live/cached/static mode, source authority, staging, immutable materializations, frozen states, schema, quality/freshness policy, institutional provenance, dependencies, and bounded reads |
| Connector layer | `govoplan-connectors` and protocol/provider modules | External protocols, endpoints, connection profiles, credentials, discovery, provider maturity/authority support, health, source-side filtering, query pushdown, and effect reconciliation |

Connectors publish versioned datasource origins. Datasources registers those
origins and presents one stable capability to Dataflow, Workflow, Reporting,
Risk Compliance, and other consumers. Consumers must not import connector
implementations or retain credentials.

The initial provider path is:

1. Connectors imports a bounded JSON/CSV snapshot and exposes it as an origin.
2. Datasources registers it as live or cached, or accepts a direct static upload
   through staging.
3. Cached refreshes and static promotions append immutable materializations.
4. Any datasource may expose a frozen state for reproducible execution evidence.
5. Dataflow stores an opaque datasource reference, state policy, and expected
   fingerprint.
6. A pinned Dataflow run may publish a complete bounded result as a new
   immutable materialization through an idempotent Datasources capability.

Database, REST/HTTP, LDAP/directory, managed file, watched-directory, feed, and
stream providers fit behind the same origin contract. Provider-specific
configuration remains in Connectors.

## Shared Definition Graph

Core owns domain-neutral graph primitives:

- nodes, typed ports, edges, and configuration field descriptors;
- node libraries and category labels;
- graph size, connectivity, cycle, and node-count constraints;
- shared backend validation and frontend connection checks.

Domain modules own their semantics:

- Dataflow provides load, combine, filter, transform, and output nodes. Its
  graph is acyclic and has one output.
- Workflow Engine provides trigger, activity, review, decision, wait,
  module-action, Dataflow, and outcome semantics. It permits governed loops and
  has exactly one trigger plus one or more outcomes. The optional Workflow
  module supplies the editor over the same native graph/BPMN language.

This division permits a shared editor shell without making Workflow a special
kind of Dataflow or leaking either module into Core.

## Current Implementation

- Core graph and datasource contracts are versioned at `0.1.0`.
- Workflow Engine owns tenant-isolated definitions, immutable revisions,
  activation pinning, module-contributed versioned baselines, runtime instances,
  governed action/effect execution, retries, waits, and reconciliation. The
  optional Workflow module exposes the reusable native BPMN graph editor.
- Datasources exposes catalogue, origins, staging, promotion, preview,
  materialization history, refresh, freeze, retirement, and producer
  publication APIs.
- Datasources WebUI exposes all current lifecycle views.
- Connectors adapts existing tabular snapshots to datasource origins.
- Dataflow consumes only Datasources catalogue/lifecycle capabilities and can
  request current, live, or latest-frozen state.
- Dataflow exposes typed graph/IR, registry-driven validation/execution/SQL
  compilation, expressions and reusable subflows, a pinned run-lifecycle
  capability, production worker boundary, and Run/Publish surface. Runs record
  lineage and intermediate artifacts and publish only complete bounded results.
- The focused composition check proves Connector origin -> Datasource ->
  pinned Dataflow run -> frozen published materialization, including replay.

## Next Slices

1. Add the provider declaration and source-authority binding used consistently
   by Connectors, Datasources, configuration packages, Ops, and Docs.
2. Add typed datasource owner/steward, legal/purpose, quality/freshness,
   classification, correction, service/process, and downstream dependency
   metadata under
   [Datasources #6](https://git.add-ideas.de/GovOPlaN/govoplan-datasources/issues/6).
3. Add SQL database and governed REST origin providers with credential-envelope
   references and bounded pushdown.
4. Add managed-file and directory origins.
5. Complete datasource quality rules, schema compatibility policy, retention, and
   promotion approvals.
6. Complete scheduled/event/API/chained Dataflow trigger governance, reusable
   template inheritance, Reporting publication, human reconciliation transforms,
   and target resource/recovery evidence for large runs.