119 lines
5.2 KiB
Markdown
119 lines
5.2 KiB
Markdown
# GovOPlaN Datasources
|
|
|
|
## Boundary
|
|
|
|
Datasources owns the governed identity and lifecycle of consumable data and
|
|
registers. It is not only a technical connection list: it is the catalogue in
|
|
which an institution explains what a dataset means, who is responsible for it,
|
|
why it may be used, how current and trustworthy it is, and which products or
|
|
decisions depend on it.
|
|
Connectors owns external protocols, connection profiles, credentials, provider
|
|
health, discovery, and source-side query pushdown.
|
|
|
|
An external system is therefore represented in two steps:
|
|
|
|
1. a connector exposes a provider-specific **origin**;
|
|
2. Datasources registers that origin as a governed live or cached datasource.
|
|
|
|
Static uploads are staged and promoted directly by Datasources. Other modules
|
|
only retain opaque datasource and materialization references.
|
|
|
|
## Modes
|
|
|
|
- `live`: reads the external origin at request time. A live source can still be
|
|
frozen to preserve evidence for a run.
|
|
- `cached`: refreshes the origin into immutable local materializations. Reads
|
|
use the current materialization unless a specific frozen revision is chosen.
|
|
- `static`: promotes supplied rows into immutable materializations and never
|
|
refreshes an external origin.
|
|
|
|
Materializations are append-only. Changing source data creates a new revision;
|
|
old revisions remain addressable for reproducibility.
|
|
|
|
The mode above describes how data is read. A separate authority declaration
|
|
states whether GovOPlaN is authoritative, an external source is authoritative,
|
|
the local state is a mirror, changes are governed in both directions, GovOPlaN
|
|
adds a governance overlay, or the entry is link-only.
|
|
|
|
## Governance Catalogue
|
|
|
|
A catalogue entry has typed, queryable governance fields for:
|
|
|
|
- owner, steward, and responsible organization/function references;
|
|
- authoritative source, source-authority mode, and transfer/data-sharing
|
|
agreement references;
|
|
- legal or organizational basis, declared purposes, and permitted consumers;
|
|
- semantic definition, schema owner, official keys, and correction procedure;
|
|
- classification, privacy constraints, retention, legal hold, and publication
|
|
rules;
|
|
- freshness objective, quality policy, validation status, known limitations,
|
|
and incident state;
|
|
- affected services/processes and dependent Dataflows, reports, controls,
|
|
decisions, and published outputs.
|
|
|
|
Authority mode, classification, publication state, owner, and responsible
|
|
organization are directly filterable catalogue columns. Lists and structured
|
|
quality/freshness rules retain typed API shapes. The governance editor is
|
|
available to datasource managers; readers see the effective explanation next
|
|
to the data preview. Each materialization captures the complete governance
|
|
state at publication time, so later ownership or policy edits do not rewrite
|
|
the explanation attached to prior evidence.
|
|
|
|
Affected-object and dependency references are also normalized into an indexed,
|
|
tenant-scoped relation table while the original JSON lists remain readable for
|
|
compatibility. Catalogue clients can filter by exact `affected_ref` and
|
|
`dependency_ref` values without receiving row access to either the datasource
|
|
or the referenced object.
|
|
|
|
Changing catalogue governance is audited. It does not rewrite source rows,
|
|
grant row access, or alter connector credentials. External and synchronized
|
|
authority modes require an explicit authoritative-source reference.
|
|
|
|
## Staging
|
|
|
|
Staging is the inspection boundary before data becomes generally consumable.
|
|
A stage records schema, row and byte counts, fingerprint, validation result,
|
|
provenance, and intended target. Promotion either creates a datasource or adds
|
|
a new immutable materialization to an existing compatible datasource.
|
|
|
|
For bounded tabular stages, the Datasource quality policy now enforces row,
|
|
field, nullability, uniqueness, numeric range, and embedded referential-set
|
|
rules. Updates receive a deterministic compatible, warning, or breaking schema
|
|
classification. Errors and breaking changes keep a stage inspectable but block
|
|
promotion; warnings remain visible and promotable. The policy hash, diagnostics,
|
|
and schema diff are retained with the promoted materialization evidence. See
|
|
[QUALITY_POLICY.md](QUALITY_POLICY.md) for the contract and its privacy bounds.
|
|
|
|
The first slice stores bounded tabular JSON/CSV stages. Future providers may
|
|
stage file references, object-store blobs, directory snapshots, or streaming
|
|
checkpoints through the same lifecycle contract.
|
|
|
|
## Consumer Contract
|
|
|
|
Consumers request:
|
|
|
|
- the current state;
|
|
- a direct live read where policy permits;
|
|
- a particular materialization; or
|
|
- the latest frozen state.
|
|
|
|
Reads are bounded and tenant-scoped. Schema and expected fingerprints allow a
|
|
Dataflow or Workflow definition to detect changed inputs before execution.
|
|
|
|
Consumers should be able to request the governance explanation and dependency
|
|
impact separately from row access. Seeing catalogue metadata must not imply
|
|
permission to read protected data.
|
|
|
|
## Next Providers
|
|
|
|
Connector providers should cover:
|
|
|
|
- SQL databases with governed query and pushdown policies;
|
|
- REST and generic HTTP requests;
|
|
- LDAP and directory services;
|
|
- managed files and watched directories;
|
|
- feeds such as RSS/Atom;
|
|
- object storage and public-sector application APIs.
|
|
|
|
Credential and connection policy remain outside this module.
|