# Datasource And Definition Graph Architecture

## Two-Layer Data Boundary

GovOPlaN separates governed data identity from external acquisition:

| Layer | Owner | Responsibilities |
| --- | --- | --- |
| Datasource layer | `govoplan-datasources` | Catalogue, tenant visibility, live/cached/static mode, staging, immutable materializations, frozen states, schema, fingerprints, provenance, and bounded reads |
| Connector layer | `govoplan-connectors` and protocol/provider modules | External protocols, endpoints, connection profiles, credentials, discovery, provider health, source-side filtering, and query pushdown |

Connectors publish versioned datasource origins. Datasources registers those
origins and presents one stable capability to Dataflow, Workflow, Reporting,
Risk Compliance, and other consumers. Consumers must not import connector
implementations or retain credentials.

The initial provider path is:

1. Connectors imports a bounded JSON/CSV snapshot and exposes it as an origin.
2. Datasources registers it as live or cached, or accepts a direct static upload
   through staging.
3. Cached refreshes and static promotions append immutable materializations.
4. Any datasource may expose a frozen state for reproducible execution evidence.
5. Dataflow stores an opaque datasource reference, state policy, and expected
   fingerprint.
6. A pinned Dataflow run may publish a complete bounded result as a new
   immutable materialization through an idempotent Datasources capability.

Database, REST/HTTP, LDAP/directory, managed file, watched-directory, feed, and
stream providers fit behind the same origin contract. Provider-specific
configuration remains in Connectors.

## Shared Definition Graph

Core owns domain-neutral graph primitives:

- nodes, typed ports, edges, and configuration field descriptors;
- node libraries and category labels;
- graph size, connectivity, cycle, and node-count constraints;
- shared backend validation and frontend connection checks.

Domain modules own their semantics:

- Dataflow provides load, combine, filter, transform, and output nodes. Its
  graph is acyclic and has one output.
- Workflow provides trigger, activity, review, decision, wait, module-action,
  Dataflow, and outcome nodes. It permits governed loops and has exactly one
  trigger plus one or more outcomes.

This division permits a shared editor shell without making Workflow a special
kind of Dataflow or leaking either module into Core.

## Current Implementation

- Core graph and datasource contracts are versioned at `0.1.0`.
- Workflow exposes a reusable graph editor, node-library discovery, validation,
  tenant-isolated definitions, immutable revisions, and activation pinning.
- Datasources exposes catalogue, origins, staging, promotion, preview,
  materialization history, refresh, freeze, retirement, and producer
  publication APIs.
- Datasources WebUI exposes all current lifecycle views.
- Connectors adapts existing tabular snapshots to datasource origins.
- Dataflow consumes only Datasources catalogue/lifecycle capabilities and can
  request current, live, or latest-frozen state.
- Dataflow exposes a pinned run-lifecycle capability and a Run/Publish surface.
  Its first synchronous runner records lineage and terminal state, publishes
  only complete bounded results, and retains output datasource/materialization
  references.
- The focused composition check proves Connector origin -> Datasource ->
  pinned Dataflow run -> frozen published materialization, including replay.

## Next Slices

1. Add persisted Workflow instances, resumable transitions, human activities,
   retry policy, and event subscriptions against pinned definition revisions.
2. Add SQL database and governed REST origin providers with credential-envelope
   references and bounded pushdown.
3. Add managed-file and directory origins.
4. Add datasource quality rules, schema compatibility policy, retention, and
   promotion approvals.
5. Add asynchronous Dataflow workers and durable artifact-backed outputs for
   runs that exceed the synchronous row/time/byte limits.