64 lines
2.3 KiB
Markdown
64 lines
2.3 KiB
Markdown
# GovOPlaN Datasources
|
|
|
|
## Boundary
|
|
|
|
Datasources owns the governed identity and lifecycle of consumable data.
|
|
Connectors owns external protocols, connection profiles, credentials, provider
|
|
health, discovery, and source-side query pushdown.
|
|
|
|
An external system is therefore represented in two steps:
|
|
|
|
1. a connector exposes a provider-specific **origin**;
|
|
2. Datasources registers that origin as a governed live or cached datasource.
|
|
|
|
Static uploads are staged and promoted directly by Datasources. Other modules
|
|
only retain opaque datasource and materialization references.
|
|
|
|
## Modes
|
|
|
|
- `live`: reads the external origin at request time. A live source can still be
|
|
frozen to preserve evidence for a run.
|
|
- `cached`: refreshes the origin into immutable local materializations. Reads
|
|
use the current materialization unless a specific frozen revision is chosen.
|
|
- `static`: promotes supplied rows into immutable materializations and never
|
|
refreshes an external origin.
|
|
|
|
Materializations are append-only. Changing source data creates a new revision;
|
|
old revisions remain addressable for reproducibility.
|
|
|
|
## Staging
|
|
|
|
Staging is the inspection boundary before data becomes generally consumable.
|
|
A stage records schema, row and byte counts, fingerprint, validation result,
|
|
provenance, and intended target. Promotion either creates a datasource or adds
|
|
a new immutable materialization to an existing compatible datasource.
|
|
|
|
The first slice stores bounded tabular JSON/CSV stages. Future providers may
|
|
stage file references, object-store blobs, directory snapshots, or streaming
|
|
checkpoints through the same lifecycle contract.
|
|
|
|
## Consumer Contract
|
|
|
|
Consumers request:
|
|
|
|
- the current state;
|
|
- a direct live read where policy permits;
|
|
- a particular materialization; or
|
|
- the latest frozen state.
|
|
|
|
Reads are bounded and tenant-scoped. Schema and expected fingerprints allow a
|
|
Dataflow or Workflow definition to detect changed inputs before execution.
|
|
|
|
## Next Providers
|
|
|
|
Connector providers should cover:
|
|
|
|
- SQL databases with governed query and pushdown policies;
|
|
- REST and generic HTTP requests;
|
|
- LDAP and directory services;
|
|
- managed files and watched directories;
|
|
- feeds such as RSS/Atom;
|
|
- object storage and public-sector application APIs.
|
|
|
|
Credential and connection policy remain outside this module.
|