feat: initialize governed datasources module

This commit is contained in:
2026-07-28 12:43:02 +02:00
commit 1cb6228442
37 changed files with 5910 additions and 0 deletions

63
docs/CONCEPT.md Normal file
View File

@@ -0,0 +1,63 @@
# GovOPlaN Datasources
## Boundary
Datasources owns the governed identity and lifecycle of consumable data.
Connectors owns external protocols, connection profiles, credentials, provider
health, discovery, and source-side query pushdown.
An external system is therefore represented in two steps:
1. a connector exposes a provider-specific **origin**;
2. Datasources registers that origin as a governed live or cached datasource.
Static uploads are staged and promoted directly by Datasources. Other modules
only retain opaque datasource and materialization references.
## Modes
- `live`: reads the external origin at request time. A live source can still be
frozen to preserve evidence for a run.
- `cached`: refreshes the origin into immutable local materializations. Reads
use the current materialization unless a specific frozen revision is chosen.
- `static`: promotes supplied rows into immutable materializations and never
refreshes an external origin.
Materializations are append-only. Changing source data creates a new revision;
old revisions remain addressable for reproducibility.
## Staging
Staging is the inspection boundary before data becomes generally consumable.
A stage records schema, row and byte counts, fingerprint, validation result,
provenance, and intended target. Promotion either creates a datasource or adds
a new immutable materialization to an existing compatible datasource.
The first slice stores bounded tabular JSON/CSV stages. Future providers may
stage file references, object-store blobs, directory snapshots, or streaming
checkpoints through the same lifecycle contract.
## Consumer Contract
Consumers request:
- the current state;
- a direct live read where policy permits;
- a particular materialization; or
- the latest frozen state.
Reads are bounded and tenant-scoped. Schema and expected fingerprints allow a
Dataflow or Workflow definition to detect changed inputs before execution.
## Next Providers
Connector providers should cover:
- SQL databases with governed query and pushdown policies;
- REST and generic HTTP requests;
- LDAP and directory services;
- managed files and watched directories;
- feeds such as RSS/Atom;
- object storage and public-sector application APIs.
Credential and connection policy remain outside this module.