feat: initialize governed datasources module
This commit is contained in:
63
docs/CONCEPT.md
Normal file
63
docs/CONCEPT.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# GovOPlaN Datasources
|
||||
|
||||
## Boundary
|
||||
|
||||
Datasources owns the governed identity and lifecycle of consumable data.
|
||||
Connectors owns external protocols, connection profiles, credentials, provider
|
||||
health, discovery, and source-side query pushdown.
|
||||
|
||||
An external system is therefore represented in two steps:
|
||||
|
||||
1. a connector exposes a provider-specific **origin**;
|
||||
2. Datasources registers that origin as a governed live or cached datasource.
|
||||
|
||||
Static uploads are staged and promoted directly by Datasources. Other modules
|
||||
only retain opaque datasource and materialization references.
|
||||
|
||||
## Modes
|
||||
|
||||
- `live`: reads the external origin at request time. A live source can still be
|
||||
frozen to preserve evidence for a run.
|
||||
- `cached`: refreshes the origin into immutable local materializations. Reads
|
||||
use the current materialization unless a specific frozen revision is chosen.
|
||||
- `static`: promotes supplied rows into immutable materializations and never
|
||||
refreshes an external origin.
|
||||
|
||||
Materializations are append-only. Changing source data creates a new revision;
|
||||
old revisions remain addressable for reproducibility.
|
||||
|
||||
## Staging
|
||||
|
||||
Staging is the inspection boundary before data becomes generally consumable.
|
||||
A stage records schema, row and byte counts, fingerprint, validation result,
|
||||
provenance, and intended target. Promotion either creates a datasource or adds
|
||||
a new immutable materialization to an existing compatible datasource.
|
||||
|
||||
The first slice stores bounded tabular JSON/CSV stages. Future providers may
|
||||
stage file references, object-store blobs, directory snapshots, or streaming
|
||||
checkpoints through the same lifecycle contract.
|
||||
|
||||
## Consumer Contract
|
||||
|
||||
Consumers request:
|
||||
|
||||
- the current state;
|
||||
- a direct live read where policy permits;
|
||||
- a particular materialization; or
|
||||
- the latest frozen state.
|
||||
|
||||
Reads are bounded and tenant-scoped. Schema and expected fingerprints allow a
|
||||
Dataflow or Workflow definition to detect changed inputs before execution.
|
||||
|
||||
## Next Providers
|
||||
|
||||
Connector providers should cover:
|
||||
|
||||
- SQL databases with governed query and pushdown policies;
|
||||
- REST and generic HTTP requests;
|
||||
- LDAP and directory services;
|
||||
- managed files and watched directories;
|
||||
- feeds such as RSS/Atom;
|
||||
- object storage and public-sector application APIs.
|
||||
|
||||
Credential and connection policy remain outside this module.
|
||||
Reference in New Issue
Block a user