Add datasource quality and schema gates

2026-08-04 11:23:14 +02:00
parent a6d5c9839d
commit 075d2fc087
16 changed files with 1305 additions and 10 deletions
@@ -0,0 +1,83 @@
+# Datasource Stage Quality Policy
+
+Datasource quality policy is a deterministic JSON contract stored in
+`governance.quality_policy`. A tabular stage inherits the current target
+Datasource policy unless the stage has its own governed definition. Validation
+runs before the stage is stored, but a failed stage remains available for
+inspection and correction through a new stage.
+
+## Contract
+
+```json
+{
+  "version": "monthly-import-v1",
+  "rules": [
+    {"id": "non-empty", "type": "row_count", "minimum": 1},
+    {"id": "columns", "type": "required_fields", "fields": ["id", "status"]},
+    {"id": "id-shape", "type": "field", "field": "id", "data_type": "integer", "nullable": false},
+    {"id": "id-present", "type": "not_null", "fields": ["id"]},
+    {"id": "id-unique", "type": "unique", "fields": ["id"]},
+    {"id": "amount-range", "type": "range", "field": "amount", "minimum": 0},
+    {"id": "known-status", "type": "referential", "field": "status", "allowed_values": ["new", "closed"]}
+  ],
+  "schema_policy": {
+    "field_added_required": "warning",
+    "field_removed": "breaking"
+  }
+}
+```
+
+Every rule has a stable `id` and may set `severity` to `error` or `warning`.
+Errors block promotion; warnings require review but leave the stage ready. The
+supported rules are:
+
+- `row_count`: optional non-negative `minimum` and `maximum`;
+- `required_fields`: fields that must exist in the detected schema;
+- `field`: expected `data_type` and/or `nullable` contract for one field;
+- `not_null`: one `field` or a `fields` list that must contain no nulls;
+- `unique`: one `field` or a composite `fields` list, with optional
+  `ignore_nulls`;
+- `range`: numeric `minimum` and/or `maximum`, with optional `allow_null`;
+- `referential`: a field and a bounded `allowed_values` set, with optional
+  `allow_null`.
+
+The bounded referential rule is intentionally local and reproducible. A future
+cross-Datasource reference rule must freeze the referenced materialization and
+perform an independent read-authorization check; it must not silently read the
+current state of another protected Datasource.
+
+Use embedded values only for non-sensitive code sets. Quality policy is
+catalogue governance metadata and can be visible to actors who are not allowed
+to read protected rows.
+
+## Schema Classification
+
+When a stage targets an existing Datasource, fields are compared by stable
+name. Defaults are:
+
+| Change | Default |
+| --- | --- |
+| nullable field added | compatible |
+| required field added | warning |
+| field removed | breaking |
+| integer widened to number, or unknown resolved | compatible |
+| other type change | breaking |
+| nullability relaxed | breaking |
+| nullability tightened | compatible |
+| existing field order changed | warning |
+
+Each key may be overridden in `schema_policy` with `compatible`, `warning`, or
+`breaking`. The highest resulting classification is the stage classification.
+A breaking classification blocks promotion.
+
+## Evidence And Privacy
+
+Validation records the policy contract version, a SHA-256 hash of the complete
+policy, rule counts, diagnostics, and the complete schema diff. Row diagnostics
+contain only affected counts and at most 25 one-based row numbers; they never
+copy field values. Promotion copies this validation object into the immutable
+materialization provenance and records the policy hash in the audit event.
+
+Approval authority, approval expiry, and retention/deletion execution remain
+separate work under `govoplan-datasources#2`. Until those contracts are added,
+no JSON flag is treated as an approval and no stage is deleted automatically.