govoplan-datasources/docs/QUALITY_POLICY.md

Datasource Stage Quality Policy

Datasource quality policy is a deterministic JSON contract stored in governance.quality_policy. A tabular stage inherits the current target Datasource policy unless the stage has its own governed definition. Validation runs before the stage is stored, but a failed stage remains available for inspection and correction through a new stage.

Contract

{
  "version": "monthly-import-v1",
  "rules": [
    {"id": "non-empty", "type": "row_count", "minimum": 1},
    {"id": "columns", "type": "required_fields", "fields": ["id", "status"]},
    {"id": "id-shape", "type": "field", "field": "id", "data_type": "integer", "nullable": false},
    {"id": "id-present", "type": "not_null", "fields": ["id"]},
    {"id": "id-unique", "type": "unique", "fields": ["id"]},
    {"id": "amount-range", "type": "range", "field": "amount", "minimum": 0},
    {"id": "known-status", "type": "referential", "field": "status", "allowed_values": ["new", "closed"]}
  ],
  "schema_policy": {
    "field_added_required": "warning",
    "field_removed": "breaking"
  }
}

Every rule has a stable id and may set severity to error or warning. Errors block promotion; warnings require review but leave the stage ready. The supported rules are:

• row_count: optional non-negative minimum and maximum;
• required_fields: fields that must exist in the detected schema;
• field: expected data_type and/or nullable contract for one field;
• not_null: one field or a fields list that must contain no nulls;
• unique: one field or a composite fields list, with optional ignore_nulls;
• range: numeric minimum and/or maximum, with optional allow_null;
• referential: a field and a bounded allowed_values set, with optional allow_null.

The bounded referential rule is intentionally local and reproducible. A future cross-Datasource reference rule must freeze the referenced materialization and perform an independent read-authorization check; it must not silently read the current state of another protected Datasource.

Use embedded values only for non-sensitive code sets. Quality policy is catalogue governance metadata and can be visible to actors who are not allowed to read protected rows.

Schema Classification

When a stage targets an existing Datasource, fields are compared by stable name. Defaults are:

Change	Default
nullable field added	compatible
required field added	warning
field removed	breaking
integer widened to number, or unknown resolved	compatible
other type change	breaking
nullability relaxed	breaking
nullability tightened	compatible
existing field order changed	warning

Each key may be overridden in schema_policy with compatible, warning, or breaking. The highest resulting classification is the stage classification. A breaking classification blocks promotion.

Evidence And Privacy

Validation records the policy contract version, a SHA-256 hash of the complete policy, rule counts, diagnostics, and the complete schema diff. Row diagnostics contain only affected counts and at most 25 one-based row numbers; they never copy field values. Promotion copies this validation object into the immutable materialization provenance and records the policy hash in the audit event.

Approval authority, approval expiry, and retention/deletion execution remain separate work under govoplan-datasources#2. Until those contracts are added, no JSON flag is treated as an approval and no stage is deleted automatically.