govoplan-datasources/docs/CONCEPT.md

GovOPlaN Datasources

Boundary

Datasources owns the governed identity and lifecycle of consumable data and registers. It is not only a technical connection list: it is the catalogue in which an institution explains what a dataset means, who is responsible for it, why it may be used, how current and trustworthy it is, and which products or decisions depend on it. Connectors owns external protocols, connection profiles, credentials, provider health, discovery, and source-side query pushdown.

An external system is therefore represented in two steps:

1. a connector exposes a provider-specific origin;
2. Datasources registers that origin as a governed live or cached datasource.

Static uploads are staged and promoted directly by Datasources. Other modules only retain opaque datasource and materialization references.

Modes

• live: reads the external origin at request time. A live source can still be frozen to preserve evidence for a run.
• cached: refreshes the origin into immutable local materializations. Reads use the current materialization unless a specific frozen revision is chosen.
• static: promotes supplied rows into immutable materializations and never refreshes an external origin.

Materializations are append-only. Changing source data creates a new revision; old revisions remain addressable for reproducibility.

The mode above describes how data is read. A separate authority declaration states whether GovOPlaN is authoritative, an external source is authoritative, the local state is a mirror, changes are governed in both directions, GovOPlaN adds a governance overlay, or the entry is link-only.

Governance Catalogue

A catalogue entry has typed, queryable governance fields for:

• owner, steward, and responsible organization/function references;
• authoritative source, source-authority mode, and transfer/data-sharing agreement references;
• legal or organizational basis, declared purposes, and permitted consumers;
• semantic definition, schema owner, official keys, and correction procedure;
• classification, privacy constraints, retention, legal hold, and publication rules;
• freshness objective, quality policy, validation status, known limitations, and incident state;
• affected services/processes and dependent Dataflows, reports, controls, decisions, and published outputs.

Authority mode, classification, publication state, owner, and responsible organization are directly filterable catalogue columns. Lists and structured quality/freshness rules retain typed API shapes. The governance editor is available to datasource managers; readers see the effective explanation next to the data preview. Each materialization captures the complete governance state at publication time, so later ownership or policy edits do not rewrite the explanation attached to prior evidence.

Affected-object and dependency references are also normalized into an indexed, tenant-scoped relation table while the original JSON lists remain readable for compatibility. Catalogue clients can filter by exact affected_ref and dependency_ref values without receiving row access to either the datasource or the referenced object.

Changing catalogue governance is audited. It does not rewrite source rows, grant row access, or alter connector credentials. External and synchronized authority modes require an explicit authoritative-source reference.

Staging

Staging is the inspection boundary before data becomes generally consumable. A stage records schema, row and byte counts, fingerprint, validation result, provenance, and intended target. Promotion either creates a datasource or adds a new immutable materialization to an existing compatible datasource.

For bounded tabular stages, the Datasource quality policy now enforces row, field, nullability, uniqueness, numeric range, and embedded referential-set rules. Updates receive a deterministic compatible, warning, or breaking schema classification. Errors and breaking changes keep a stage inspectable but block promotion; warnings remain visible and promotable. The policy hash, diagnostics, and schema diff are retained with the promoted materialization evidence. See QUALITY_POLICY.md for the contract and its privacy bounds.

The first slice stores bounded tabular JSON/CSV stages. Future providers may stage file references, object-store blobs, directory snapshots, or streaming checkpoints through the same lifecycle contract.

Consumer Contract

Consumers request:

• the current state;
• a direct live read where policy permits;
• a particular materialization; or
• the latest frozen state.

Reads are bounded and tenant-scoped. Schema and expected fingerprints allow a Dataflow or Workflow definition to detect changed inputs before execution.

Consumers should be able to request the governance explanation and dependency impact separately from row access. Seeing catalogue metadata must not imply permission to read protected data.

Next Providers

Connector providers should cover:

• SQL databases with governed query and pushdown policies;
• REST and generic HTTP requests;
• LDAP and directory services;
• managed files and watched directories;
• feeds such as RSS/Atom;
• object storage and public-sector application APIs.

Credential and connection policy remain outside this module.