# Datasource Quality Policy

Datasource quality policy is a deterministic JSON contract stored in
`governance.quality_policy`. A tabular stage or producer publication inherits
the current target Datasource policy unless it supplies its own governed
definition. Stage validation runs before the stage is stored, but a failed
stage remains available for inspection and correction through a new stage.

## Contract

```json
{
  "version": "monthly-import-v1",
  "rules": [
    {"id": "non-empty", "type": "row_count", "minimum": 1},
    {"id": "columns", "type": "required_fields", "fields": ["id", "status"]},
    {"id": "id-shape", "type": "field", "field": "id", "data_type": "integer", "nullable": false},
    {"id": "id-present", "type": "not_null", "fields": ["id"]},
    {"id": "id-unique", "type": "unique", "fields": ["id"]},
    {"id": "amount-range", "type": "range", "field": "amount", "minimum": 0},
    {"id": "known-status", "type": "referential", "field": "status", "allowed_values": ["new", "closed"]}
  ],
  "schema_policy": {
    "field_added_required": "warning",
    "field_removed": "breaking"
  }
}
```

Every rule has a stable `id` and may set `severity` to `error` or `warning`.
Errors block promotion; warnings require review but leave the stage ready. The
supported rules are:

- `row_count`: optional non-negative `minimum` and `maximum`;
- `required_fields`: fields that must exist in the detected schema;
- `field`: expected `data_type` and/or `nullable` contract for one field;
- `not_null`: one `field` or a `fields` list that must contain no nulls;
- `unique`: one `field` or a composite `fields` list, with optional
  `ignore_nulls`;
- `range`: numeric `minimum` and/or `maximum`, with optional `allow_null`;
- `referential`: a field and a bounded `allowed_values` set, with optional
  `allow_null`.

The bounded referential rule is intentionally local and reproducible. A future
cross-Datasource reference rule must freeze the referenced materialization and
perform an independent read-authorization check; it must not silently read the
current state of another protected Datasource.

Use embedded values only for non-sensitive code sets. Quality policy is
catalogue governance metadata and can be visible to actors who are not allowed
to read protected rows.

## Schema Classification

When a stage targets an existing Datasource, fields are compared by stable
name. Defaults are:

| Change | Default |
| --- | --- |
| nullable field added | compatible |
| required field added | warning |
| field removed | breaking |
| integer widened to number, or unknown resolved | compatible |
| other type change | breaking |
| nullability relaxed | breaking |
| nullability tightened | compatible |
| existing field order changed | warning |

Each key may be overridden in `schema_policy` with `compatible`, `warning`, or
`breaking`. The highest resulting classification is the stage classification.
A breaking classification blocks promotion.

## Evidence And Privacy

Validation records the policy contract version, a SHA-256 hash of the complete
policy, rule counts, diagnostics, and the complete schema diff. Row diagnostics
contain only affected counts and at most 25 one-based row numbers; they never
copy field values. Promotion copies this validation object into the immutable
materialization provenance and records the policy hash in the audit event.

Producer publication uses the same gate before any catalogue target,
materialization, or publication record is persisted. A rejected output has no
partial catalogue effect. Successful output materializations retain the exact
validation result, policy version/hash, and schema classification in
`provenance.publication_validation`; the publication record retains the same
evidence for operational inspection. Dataflow, Workflow, and Reporting can
therefore consume an immutable output reference without re-running a possibly
changed quality policy. Publication also emits a transactional
`datasource.publication.published` audit/platform event; an enabled Audit
module stores it in the durable outbox, while reduced installations deliver it
through Core after commit.

PostgreSQL deployments serialize publication attempts by tenant, producer, and
idempotency key with a transaction-scoped advisory lock before replay lookup.
This makes concurrent retries from separate API or worker nodes converge on the
same publication and materialization rather than relying on a late uniqueness
failure after output rows have already been persisted.

## Durable artifact publications

Outputs larger than the inline row and byte limits use an immutable artifact
reference. The reference pins its backend and locator together with SHA-256
checksum, schema, datasource fingerprint, row count, byte count, media type,
and optional resume checkpoint. Datasources persists that reference as the
materialization payload and asks the installed Core-contract artifact backend
to verify it before creating catalogue state. Reads remain bounded and are
re-authorized by Datasources before reaching the backend.

Schema rules are evaluated by Datasources. Content-level rules such as
uniqueness or range require producer evidence bound to the exact payload
checksum and current quality-policy hash, including all evaluated rule IDs.
Missing or explicitly deferred evidence produces a `review_required`
publication and an immutable, addressable materialization, but it does not
replace the Datasource's current state. Valid warnings produce
`published_with_warnings`; failed evidence blocks the publication without a
catalogue side effect. These terminal states are preserved for Dataflow and
Workflow handoffs instead of being collapsed into generic success.

Approval authority, approval expiry, and retention/deletion execution remain
separate work under `govoplan-datasources#2`. Until those contracts are added,
no JSON flag is treated as an approval and no stage is deleted automatically.