govoplan-datasources/docs/QUALITY_POLICY.md

Datasource Quality Policy

Datasource quality policy is a deterministic JSON contract stored in governance.quality_policy. A tabular stage or producer publication inherits the current target Datasource policy unless it supplies its own governed definition. Stage validation runs before the stage is stored, but a failed stage remains available for inspection and correction through a new stage.

Contract

{
  "version": "monthly-import-v1",
  "rules": [
    {"id": "non-empty", "type": "row_count", "minimum": 1},
    {"id": "columns", "type": "required_fields", "fields": ["id", "status"]},
    {"id": "id-shape", "type": "field", "field": "id", "data_type": "integer", "nullable": false},
    {"id": "id-present", "type": "not_null", "fields": ["id"]},
    {"id": "id-unique", "type": "unique", "fields": ["id"]},
    {"id": "amount-range", "type": "range", "field": "amount", "minimum": 0},
    {"id": "known-status", "type": "referential", "field": "status", "allowed_values": ["new", "closed"]}
  ],
  "schema_policy": {
    "field_added_required": "warning",
    "field_removed": "breaking"
  }
}

Every rule has a stable id and may set severity to error or warning. Errors block promotion; warnings require review but leave the stage ready. The supported rules are:

• row_count: optional non-negative minimum and maximum;
• required_fields: fields that must exist in the detected schema;
• field: expected data_type and/or nullable contract for one field;
• not_null: one field or a fields list that must contain no nulls;
• unique: one field or a composite fields list, with optional ignore_nulls;
• range: numeric minimum and/or maximum, with optional allow_null;
• referential: a field and a bounded allowed_values set, with optional allow_null.

The bounded referential rule is intentionally local and reproducible. A future cross-Datasource reference rule must freeze the referenced materialization and perform an independent read-authorization check; it must not silently read the current state of another protected Datasource.

Use embedded values only for non-sensitive code sets. Quality policy is catalogue governance metadata and can be visible to actors who are not allowed to read protected rows.

Schema Classification

When a stage targets an existing Datasource, fields are compared by stable name. Defaults are:

Change	Default
nullable field added	compatible
required field added	warning
field removed	breaking
integer widened to number, or unknown resolved	compatible
other type change	breaking
nullability relaxed	breaking
nullability tightened	compatible
existing field order changed	warning

Each key may be overridden in schema_policy with compatible, warning, or breaking. The highest resulting classification is the stage classification. A breaking classification blocks promotion.

Evidence And Privacy

Validation records the policy contract version, a SHA-256 hash of the complete policy, rule counts, diagnostics, and the complete schema diff. Row diagnostics contain only affected counts and at most 25 one-based row numbers; they never copy field values. Promotion copies this validation object into the immutable materialization provenance and records the policy hash in the audit event.

Producer publication uses the same gate before any catalogue target, materialization, or publication record is persisted. A rejected output has no partial catalogue effect. Successful output materializations retain the exact validation result, policy version/hash, and schema classification in provenance.publication_validation; the publication record retains the same evidence for operational inspection. Dataflow, Workflow, and Reporting can therefore consume an immutable output reference without re-running a possibly changed quality policy. Publication also emits a transactional datasource.publication.published audit/platform event; an enabled Audit module stores it in the durable outbox, while reduced installations deliver it through Core after commit.

PostgreSQL deployments serialize publication attempts by tenant, producer, and idempotency key with a transaction-scoped advisory lock before replay lookup. This makes concurrent retries from separate API or worker nodes converge on the same publication and materialization rather than relying on a late uniqueness failure after output rows have already been persisted.

Approval authority, approval expiry, and retention/deletion execution remain separate work under govoplan-datasources#2. Until those contracts are added, no JSON flag is treated as an approval and no stage is deleted automatically.