feat: implement governed datasource catalogue metadata

This commit is contained in:
2026-08-01 17:48:28 +02:00
parent 8da8035693
commit 98d93e707f
14 changed files with 1382 additions and 12 deletions
+48 -1
View File
@@ -2,7 +2,11 @@
## Boundary
Datasources owns the governed identity and lifecycle of consumable data.
Datasources owns the governed identity and lifecycle of consumable data and
registers. It is not only a technical connection list: it is the catalogue in
which an institution explains what a dataset means, who is responsible for it,
why it may be used, how current and trustworthy it is, and which products or
decisions depend on it.
Connectors owns external protocols, connection profiles, credentials, provider
health, discovery, and source-side query pushdown.
@@ -26,6 +30,45 @@ only retain opaque datasource and materialization references.
Materializations are append-only. Changing source data creates a new revision;
old revisions remain addressable for reproducibility.
The mode above describes how data is read. A separate authority declaration
states whether GovOPlaN is authoritative, an external source is authoritative,
the local state is a mirror, changes are governed in both directions, GovOPlaN
adds a governance overlay, or the entry is link-only.
## Governance Catalogue
A catalogue entry has typed, queryable governance fields for:
- owner, steward, and responsible organization/function references;
- authoritative source, source-authority mode, and transfer/data-sharing
agreement references;
- legal or organizational basis, declared purposes, and permitted consumers;
- semantic definition, schema owner, official keys, and correction procedure;
- classification, privacy constraints, retention, legal hold, and publication
rules;
- freshness objective, quality policy, validation status, known limitations,
and incident state;
- affected services/processes and dependent Dataflows, reports, controls,
decisions, and published outputs.
Authority mode, classification, publication state, owner, and responsible
organization are directly filterable catalogue columns. Lists and structured
quality/freshness rules retain typed API shapes. The governance editor is
available to datasource managers; readers see the effective explanation next
to the data preview. Each materialization captures the complete governance
state at publication time, so later ownership or policy edits do not rewrite
the explanation attached to prior evidence.
Affected-object and dependency references are also normalized into an indexed,
tenant-scoped relation table while the original JSON lists remain readable for
compatibility. Catalogue clients can filter by exact `affected_ref` and
`dependency_ref` values without receiving row access to either the datasource
or the referenced object.
Changing catalogue governance is audited. It does not rewrite source rows,
grant row access, or alter connector credentials. External and synchronized
authority modes require an explicit authoritative-source reference.
## Staging
Staging is the inspection boundary before data becomes generally consumable.
@@ -49,6 +92,10 @@ Consumers request:
Reads are bounded and tenant-scoped. Schema and expected fingerprints allow a
Dataflow or Workflow definition to detect changed inputs before execution.
Consumers should be able to request the governance explanation and dependency
impact separately from row access. Seeing catalogue metadata must not imply
permission to read protected data.
## Next Providers
Connector providers should cover: