feat: integrate datasource and workflow modules
Some checks failed
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled

This commit is contained in:
2026-07-28 12:45:05 +02:00
parent 603e07cec5
commit 163b35c0af
13 changed files with 262 additions and 8 deletions

View File

@@ -1279,8 +1279,9 @@ them; they do not block the product vision today.
- Which exact HIS/CampusOnline interfaces, student-statistics fields and
official keys, accepted calculation, freeze/correction policy, privacy
profile, and drill-down level should define the first analytical data product?
- After the first source-to-report proof, do repeated source/dataflow contracts
justify separate `govoplan-datasources` and `govoplan-dataflow` modules?
- Which database, REST, directory, and managed-file providers should follow the
implemented separation of `govoplan-connectors`, `govoplan-datasources`, and
`govoplan-dataflow`, and which quality/promotion policy should prove it first?
- Which collaborative editor should be the first target, and should its first
accepted experience emphasize concurrent editing, controlled check-out, or
both?

View File

@@ -0,0 +1,71 @@
# Datasource And Definition Graph Architecture
## Two-Layer Data Boundary
GovOPlaN separates governed data identity from external acquisition:
| Layer | Owner | Responsibilities |
| --- | --- | --- |
| Datasource layer | `govoplan-datasources` | Catalogue, tenant visibility, live/cached/static mode, staging, immutable materializations, frozen states, schema, fingerprints, provenance, and bounded reads |
| Connector layer | `govoplan-connectors` and protocol/provider modules | External protocols, endpoints, connection profiles, credentials, discovery, provider health, source-side filtering, and query pushdown |
Connectors publish versioned datasource origins. Datasources registers those
origins and presents one stable capability to Dataflow, Workflow, Reporting,
Risk Compliance, and other consumers. Consumers must not import connector
implementations or retain credentials.
The initial provider path is:
1. Connectors imports a bounded JSON/CSV snapshot and exposes it as an origin.
2. Datasources registers it as live or cached, or accepts a direct static upload
through staging.
3. Cached refreshes and static promotions append immutable materializations.
4. Any datasource may expose a frozen state for reproducible execution evidence.
5. Dataflow stores an opaque datasource reference, state policy, and expected
fingerprint.
Database, REST/HTTP, LDAP/directory, managed file, watched-directory, feed, and
stream providers fit behind the same origin contract. Provider-specific
configuration remains in Connectors.
## Shared Definition Graph
Core owns domain-neutral graph primitives:
- nodes, typed ports, edges, and configuration field descriptors;
- node libraries and category labels;
- graph size, connectivity, cycle, and node-count constraints;
- shared backend validation and frontend connection checks.
Domain modules own their semantics:
- Dataflow provides load, combine, filter, transform, and output nodes. Its
graph is acyclic and has one output.
- Workflow provides trigger, activity, review, decision, wait, module-action,
Dataflow, and outcome nodes. It permits governed loops and has exactly one
trigger plus one or more outcomes.
This division permits a shared editor shell without making Workflow a special
kind of Dataflow or leaking either module into Core.
## Current Implementation
- Core graph and datasource contracts are versioned at `0.1.0`.
- Workflow exposes node-library discovery and definition validation APIs.
- Datasources exposes catalogue, origins, staging, promotion, preview,
materialization history, refresh, freeze, and retirement APIs.
- Datasources WebUI exposes all current lifecycle views.
- Connectors adapts existing tabular snapshots to datasource origins.
- Dataflow consumes only Datasources catalogue/lifecycle capabilities and can
request current, live, or latest-frozen state.
## Next Slices
1. Persist and version Workflow definitions, then reuse the graph UI shell.
2. Add SQL database and governed REST origin providers with credential-envelope
references and bounded pushdown.
3. Add managed-file and directory origins.
4. Add datasource quality rules, schema compatibility policy, retention, and
promotion approvals.
5. Publish Dataflow outputs as new materializations and expose run lifecycle to
Workflow.

View File

@@ -445,8 +445,9 @@ that first needs them:
- exact HIS/CampusOnline source endpoints, student-statistics keys and accepted
calculation, freeze/correction policy, privacy profile, and permitted
drill-down level;
- whether repeated data-source and dataflow contracts justify separate
`govoplan-datasources` and `govoplan-dataflow` modules after the Stage 3 proof;
- datasource provider selection and quality/promotion policy; the architecture
now separates `govoplan-datasources` lifecycle from `govoplan-connectors`
acquisition and `govoplan-dataflow` transformation;
- first collaborative editor/provider and whether the first UX is concurrent
editing, controlled check-out, or both; and
- first Records/archive target and approval/signature assurance level.

View File

@@ -29,6 +29,8 @@ Generated from `repositories.json`. Use that JSON file as the machine-readable s
| `govoplan-consultation` | `domain` | `../govoplan-consultation` | [govoplan-consultation](https://git.add-ideas.de/GovOPlaN/govoplan-consultation) |
| `govoplan-contracts` | `domain` | `../govoplan-contracts` | [govoplan-contracts](https://git.add-ideas.de/GovOPlaN/govoplan-contracts) |
| `govoplan-dashboard` | `platform` | `../govoplan-dashboard` | [govoplan-dashboard](https://git.add-ideas.de/GovOPlaN/govoplan-dashboard) |
| `govoplan-dataflow` | `platform` | `../govoplan-dataflow` | [govoplan-dataflow](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow) |
| `govoplan-datasources` | `platform` | `../govoplan-datasources` | [govoplan-datasources](https://git.add-ideas.de/GovOPlaN/govoplan-datasources) |
| `govoplan-dms` | `domain` | `../govoplan-dms` | [govoplan-dms](https://git.add-ideas.de/GovOPlaN/govoplan-dms) |
| `govoplan-dist-lists` | `domain` | `../govoplan-dist-lists` | [govoplan-dist-lists](https://git.add-ideas.de/GovOPlaN/govoplan-dist-lists) |
| `govoplan-docs` | `platform` | `../govoplan-docs` | [govoplan-docs](https://git.add-ideas.de/GovOPlaN/govoplan-docs) |