[Feature] Extend Datasources into a governed data and register catalogue #6

New Issue

2026-08-01T00:00:53+02:00

zemion commented

2026-08-01 00:00:53 +02:00

Outcome

Extend the implemented live/cached/static, staging, immutable-materialization, and publication foundation into the institution's governed catalogue for data and registers.

Parent: GovOPlaN/govoplan#29
Parent module epic: #1

Scope

Add typed, queryable fields and APIs for owner/steward/responsible organization/function, authoritative source and source-authority mode, legal/organizational basis, declared purposes, semantic definition/schema owner/official keys, classification/privacy/retention/hold/publication, transfer agreement, freshness and quality policy, known limits/incidents, correction procedure, affected services/processes, and dependent flows/reports/controls/decisions/outputs.

Guardrails

Catalogue metadata visibility does not grant row access.
Connector profiles, credentials, protocol health, and query pushdown remain in Connectors.
Dataflow owns transformations; Reporting owns report execution/presentation.
Existing provenance/metadata JSON remains readable while policy- and impact-relevant fields migrate to typed schemas.

Acceptance criteria

Migration and API support compatibility for existing catalogue entries.
Policy-aware field and row access remains independent from catalogue discovery.
Dependency and impact queries are bounded and tenant-safe.
Materializations preserve the effective governance snapshot needed for reproducibility.
UI and Docs explain authority, freshness, quality, purpose, owner, and limitations.

## Outcome Extend the implemented live/cached/static, staging, immutable-materialization, and publication foundation into the institution's governed catalogue for data and registers. Parent: https://git.add-ideas.de/GovOPlaN/govoplan/issues/29 Parent module epic: https://git.add-ideas.de/GovOPlaN/govoplan-datasources/issues/1 ## Scope Add typed, queryable fields and APIs for owner/steward/responsible organization/function, authoritative source and source-authority mode, legal/organizational basis, declared purposes, semantic definition/schema owner/official keys, classification/privacy/retention/hold/publication, transfer agreement, freshness and quality policy, known limits/incidents, correction procedure, affected services/processes, and dependent flows/reports/controls/decisions/outputs. ## Guardrails - Catalogue metadata visibility does not grant row access. - Connector profiles, credentials, protocol health, and query pushdown remain in Connectors. - Dataflow owns transformations; Reporting owns report execution/presentation. - Existing provenance/metadata JSON remains readable while policy- and impact-relevant fields migrate to typed schemas. ## Acceptance criteria - Migration and API support compatibility for existing catalogue entries. - Policy-aware field and row access remains independent from catalogue discovery. - Dependency and impact queries are bounded and tenant-safe. - Materializations preserve the effective governance snapshot needed for reproducibility. - UI and Docs explain authority, freshness, quality, purpose, owner, and limitations.

zemion added the module/policy module/connectors module/dataflow module/datasources module/reporting area/governance area/db

area/api labels 2026-08-01 00:00:53 +02:00

zemion referenced this issue from GovOPlaN/govoplan

2026-08-01 00:00:54 +02:00

[Epic] Adopt the institutional governance target architecture #29

zemion referenced this issue

2026-08-01 00:01:55 +02:00

[Epic] Deliver the governed datasource catalogue and lifecycle #1

zemion commented

2026-08-01 03:32:40 +02:00

Codex State: done

Summary

Added typed catalogue governance for owner/steward/responsibility, authority, legal basis, purposes, schema, classification/privacy/retention, freshness/quality, limits/incidents, correction, transfer, impact, and dependencies.
Materializations now freeze the effective governance snapshot, while legacy metadata JSON remains readable.
Indexed exact affected/dependency references support bounded tenant-scoped impact lookup and are synchronized on updates.
API filters and the WebUI expose authority, freshness, quality, purpose, ownership, and limitations without coupling catalogue discovery to row access.

Changed Files

src/govoplan_datasources/backend/db/models.py
src/govoplan_datasources/backend/services/catalog.py
src/govoplan_datasources/backend/api/v1/routes.py
alembic/versions/b8d2f5a0c3e7_v0114_governance_reference_index.py
docs/DATASOURCE_LIFECYCLE.md

Verification

Datasources backend suite: 17 passed, 1 skipped
Datasources WebUI TypeScript check: passed

## Codex State: done ### Summary - Added typed catalogue governance for owner/steward/responsibility, authority, legal basis, purposes, schema, classification/privacy/retention, freshness/quality, limits/incidents, correction, transfer, impact, and dependencies. - Materializations now freeze the effective governance snapshot, while legacy metadata JSON remains readable. - Indexed exact affected/dependency references support bounded tenant-scoped impact lookup and are synchronized on updates. - API filters and the WebUI expose authority, freshness, quality, purpose, ownership, and limitations without coupling catalogue discovery to row access. ### Changed Files - `src/govoplan_datasources/backend/db/models.py` - `src/govoplan_datasources/backend/services/catalog.py` - `src/govoplan_datasources/backend/api/v1/routes.py` - `alembic/versions/b8d2f5a0c3e7_v0114_governance_reference_index.py` - `docs/DATASOURCE_LIFECYCLE.md` ### Verification - `Datasources backend suite: 17 passed, 1 skipped` - `Datasources WebUI TypeScript check: passed`

zemion closed this issue

2026-08-01 03:32:40 +02:00

zemion removed the

status

ready

codex/ready labels 2026-08-01 03:36:50 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-datasources#6