[Feature] Add datasource quality gates, schema policy, approvals, and retention #2

New Issue

2026-07-28T12:48:19+02:00

zemion commented

2026-07-28 12:48:19 +02:00

Parent: #1

Add governed lifecycle policy beyond the first bounded staging implementation.

Acceptance criteria:

configurable row, field, uniqueness, nullability, range, and referential quality rules
explicit compatible, warning, and breaking schema-change classifications
previewable promotion diagnostics and diffs
optional approval gate before a stage or refresh becomes current
retention rules for stages, ordinary materializations, and frozen evidence
immutable audit evidence for policy version, validation result, approver, and promotion
policy-aware behavior remains usable without Access/Policy modules, with reduced local defaults

Parent: https://git.add-ideas.de/GovOPlaN/govoplan-datasources/issues/1 Add governed lifecycle policy beyond the first bounded staging implementation. Acceptance criteria: - configurable row, field, uniqueness, nullability, range, and referential quality rules - explicit compatible, warning, and breaking schema-change classifications - previewable promotion diagnostics and diffs - optional approval gate before a stage or refresh becomes current - retention rules for stages, ordinary materializations, and frozen evidence - immutable audit evidence for policy version, validation result, approver, and promotion - policy-aware behavior remains usable without Access/Policy modules, with reduced local defaults

zemion added the module/datasources area/api area/governance area/db codex/ready

labels 2026-07-28 12:48:19 +02:00

zemion referenced this issue from GovOPlaN/govoplan-dataflow

2026-07-28 12:48:20 +02:00

[Feature] Consume connector-backed tabular sources and schema discovery #6

zemion commented

2026-08-04 11:23:43 +02:00

Codex State: progress

Summary

Implemented deterministic tabular stage quality gates for row count, required fields, field type/nullability, not-null, uniqueness, numeric range, and bounded referential value sets.
Implemented explicit compatible/warning/breaking schema diffs with governed per-change overrides; errors and breaking changes remain inspectable but cannot be promoted.
Added typed API responses and Staging UI diagnostics. Validation evidence contains bounded row numbers but no values, and successful promotion copies the policy hash and full result into immutable materialization provenance.

Changed Files

govoplan-datasources@075d2fc

Verification

pytest -q: 24 passed, 1 PostgreSQL environment test skipped
ruff check src tests: passed
npm run typecheck: passed
core npm run build and bundle budgets: passed
check-focused.sh: passed

Next / Blocked

Define the optional approval authority, separation-of-duties and expiry contract before implementing approval-gated promotion.
Define retention/hold precedence and deletion evidence before automated cleanup of stages, ordinary materializations, and frozen evidence.
Cross-Datasource referential checks require a frozen reference and independent row-read authorization; current rules deliberately support only embedded non-sensitive code sets.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Implemented deterministic tabular stage quality gates for row count, required fields, field type/nullability, not-null, uniqueness, numeric range, and bounded referential value sets. - Implemented explicit compatible/warning/breaking schema diffs with governed per-change overrides; errors and breaking changes remain inspectable but cannot be promoted. - Added typed API responses and Staging UI diagnostics. Validation evidence contains bounded row numbers but no values, and successful promotion copies the policy hash and full result into immutable materialization provenance. ### Changed Files - `govoplan-datasources@075d2fc` ### Verification - `pytest -q: 24 passed, 1 PostgreSQL environment test skipped` - `ruff check src tests: passed` - `npm run typecheck: passed` - `core npm run build and bundle budgets: passed` - `check-focused.sh: passed` ### Next / Blocked - Define the optional approval authority, separation-of-duties and expiry contract before implementing approval-gated promotion. - Define retention/hold precedence and deletion evidence before automated cleanup of stages, ordinary materializations, and frozen evidence. - Cross-Datasource referential checks require a frozen reference and independent row-read authorization; current rules deliberately support only embedded non-sensitive code sets. Suggested status label: `status/in-progress`

zemion commented

2026-08-22 20:01:41 +02:00

Implemented and released in Datasources v0.1.21 (7a7654c), with the additive Core lifecycle contract in Core v0.1.36 (c6f6faf) and workspace composition v0.1.36 (6464032).

Delivered:

configurable row/field/uniqueness/null/range/referential quality rules and compatible/warning/breaking schema diagnostics remain enforced on staging and publication
versioned optional stage/refresh approval policies with 1-5 distinct approvers, separation of duties, expiry, attributable reasons, optimistic policy/content digests, rejection and idempotent replay
approval-aware connector refresh staging; direct refresh fails closed whenever approval is required
independently configurable retention for transient stages, ordinary materializations, and frozen evidence
read-only hashed retention previews, explicit selected apply, current/legal-hold/pending-approval/publication blockers, and payload disposal tombstones
immutable hash-chained lifecycle evidence for validation, approval/rejection, promotion, and retention; minimized DSAR discovery preserves operator-attribution evidence
local reduced-mode defaults remain usable without optional Access or Policy providers
UI flows use shared semantic page/action/dialog primitives, including shared Dialog size=wide and destructive confirmation separation
administrator/user documentation and migration/manifest contracts updated

Verification:

Datasources: 56 tests passed, 2 skipped; Ruff and TypeScript clean
manifest registry: 68/68 manifests passed
full tools/checks/check-focused.sh: passed, including backend integration suites, the new migration chain, 63 WebUI module permutations, full-product bundle budget, 8 Playwright conformance tests, and module-specific frontend checks

The separate release-composition audit still identifies pre-existing stale WebUI release refs in unrelated Audit/Campaign/Docs/IDM/Mail/Ops/Policy packages; this slice did not introduce or modify those refs.

Implemented and released in Datasources v0.1.21 (`7a7654c`), with the additive Core lifecycle contract in Core v0.1.36 (`c6f6faf`) and workspace composition v0.1.36 (`6464032`). Delivered: - configurable row/field/uniqueness/null/range/referential quality rules and compatible/warning/breaking schema diagnostics remain enforced on staging and publication - versioned optional stage/refresh approval policies with 1-5 distinct approvers, separation of duties, expiry, attributable reasons, optimistic policy/content digests, rejection and idempotent replay - approval-aware connector refresh staging; direct refresh fails closed whenever approval is required - independently configurable retention for transient stages, ordinary materializations, and frozen evidence - read-only hashed retention previews, explicit selected apply, current/legal-hold/pending-approval/publication blockers, and payload disposal tombstones - immutable hash-chained lifecycle evidence for validation, approval/rejection, promotion, and retention; minimized DSAR discovery preserves operator-attribution evidence - local reduced-mode defaults remain usable without optional Access or Policy providers - UI flows use shared semantic page/action/dialog primitives, including shared `Dialog size=wide` and destructive confirmation separation - administrator/user documentation and migration/manifest contracts updated Verification: - Datasources: 56 tests passed, 2 skipped; Ruff and TypeScript clean - manifest registry: 68/68 manifests passed - full `tools/checks/check-focused.sh`: passed, including backend integration suites, the new migration chain, 63 WebUI module permutations, full-product bundle budget, 8 Playwright conformance tests, and module-specific frontend checks The separate release-composition audit still identifies pre-existing stale WebUI release refs in unrelated Audit/Campaign/Docs/IDM/Mail/Ops/Policy packages; this slice did not introduce or modify those refs.

zemion closed this issue

2026-08-22 20:01:41 +02:00

zemion referenced this issue

2026-08-22 20:01:42 +02:00

[Epic] Deliver the governed datasource catalogue and lifecycle #1

zemion referenced this issue

2026-08-22 20:02:28 +02:00

[Epic] Deliver the governed datasource catalogue and lifecycle #1

Sign in to join this conversation.