GovOPlaN Datasources

Boundary

Datasources owns the governed identity and lifecycle of consumable data. Connectors owns external protocols, connection profiles, credentials, provider health, discovery, and source-side query pushdown.

An external system is therefore represented in two steps:

a connector exposes a provider-specific origin;
Datasources registers that origin as a governed live or cached datasource.

Static uploads are staged and promoted directly by Datasources. Other modules only retain opaque datasource and materialization references.

Modes

live: reads the external origin at request time. A live source can still be frozen to preserve evidence for a run.
cached: refreshes the origin into immutable local materializations. Reads use the current materialization unless a specific frozen revision is chosen.
static: promotes supplied rows into immutable materializations and never refreshes an external origin.

Materializations are append-only. Changing source data creates a new revision; old revisions remain addressable for reproducibility.

Staging

Staging is the inspection boundary before data becomes generally consumable. A stage records schema, row and byte counts, fingerprint, validation result, provenance, and intended target. Promotion either creates a datasource or adds a new immutable materialization to an existing compatible datasource.

The first slice stores bounded tabular JSON/CSV stages. Future providers may stage file references, object-store blobs, directory snapshots, or streaming checkpoints through the same lifecycle contract.

Consumer Contract

Consumers request:

the current state;
a direct live read where policy permits;
a particular materialization; or
the latest frozen state.

Reads are bounded and tenant-scoped. Schema and expected fingerprints allow a Dataflow or Workflow definition to detect changed inputs before execution.

Next Providers

Connector providers should cover:

SQL databases with governed query and pushdown policies;
REST and generic HTTP requests;
LDAP and directory services;
managed files and watched directories;
feeds such as RSS/Atom;
object storage and public-sector application APIs.

Credential and connection policy remain outside this module.

2.3 KiB Raw Blame History