122 lines
5.8 KiB
Markdown
122 lines
5.8 KiB
Markdown
# Datasource Quality Policy
|
|
|
|
Datasource quality policy is a deterministic JSON contract stored in
|
|
`governance.quality_policy`. A tabular stage or producer publication inherits
|
|
the current target Datasource policy unless it supplies its own governed
|
|
definition. Stage validation runs before the stage is stored, but a failed
|
|
stage remains available for inspection and correction through a new stage.
|
|
|
|
## Contract
|
|
|
|
```json
|
|
{
|
|
"version": "monthly-import-v1",
|
|
"rules": [
|
|
{"id": "non-empty", "type": "row_count", "minimum": 1},
|
|
{"id": "columns", "type": "required_fields", "fields": ["id", "status"]},
|
|
{"id": "id-shape", "type": "field", "field": "id", "data_type": "integer", "nullable": false},
|
|
{"id": "id-present", "type": "not_null", "fields": ["id"]},
|
|
{"id": "id-unique", "type": "unique", "fields": ["id"]},
|
|
{"id": "amount-range", "type": "range", "field": "amount", "minimum": 0},
|
|
{"id": "known-status", "type": "referential", "field": "status", "allowed_values": ["new", "closed"]}
|
|
],
|
|
"schema_policy": {
|
|
"field_added_required": "warning",
|
|
"field_removed": "breaking"
|
|
}
|
|
}
|
|
```
|
|
|
|
Every rule has a stable `id` and may set `severity` to `error` or `warning`.
|
|
Errors block promotion; warnings require review but leave the stage ready. The
|
|
supported rules are:
|
|
|
|
- `row_count`: optional non-negative `minimum` and `maximum`;
|
|
- `required_fields`: fields that must exist in the detected schema;
|
|
- `field`: expected `data_type` and/or `nullable` contract for one field;
|
|
- `not_null`: one `field` or a `fields` list that must contain no nulls;
|
|
- `unique`: one `field` or a composite `fields` list, with optional
|
|
`ignore_nulls`;
|
|
- `range`: numeric `minimum` and/or `maximum`, with optional `allow_null`;
|
|
- `referential`: a field and a bounded `allowed_values` set, with optional
|
|
`allow_null`.
|
|
|
|
The bounded referential rule is intentionally local and reproducible. A future
|
|
cross-Datasource reference rule must freeze the referenced materialization and
|
|
perform an independent read-authorization check; it must not silently read the
|
|
current state of another protected Datasource.
|
|
|
|
Use embedded values only for non-sensitive code sets. Quality policy is
|
|
catalogue governance metadata and can be visible to actors who are not allowed
|
|
to read protected rows.
|
|
|
|
## Schema Classification
|
|
|
|
When a stage targets an existing Datasource, fields are compared by stable
|
|
name. Defaults are:
|
|
|
|
| Change | Default |
|
|
| --- | --- |
|
|
| nullable field added | compatible |
|
|
| required field added | warning |
|
|
| field removed | breaking |
|
|
| integer widened to number, or unknown resolved | compatible |
|
|
| other type change | breaking |
|
|
| nullability relaxed | breaking |
|
|
| nullability tightened | compatible |
|
|
| existing field order changed | warning |
|
|
|
|
Each key may be overridden in `schema_policy` with `compatible`, `warning`, or
|
|
`breaking`. The highest resulting classification is the stage classification.
|
|
A breaking classification blocks promotion.
|
|
|
|
## Evidence And Privacy
|
|
|
|
Validation records the policy contract version, a SHA-256 hash of the complete
|
|
policy, rule counts, diagnostics, and the complete schema diff. Row diagnostics
|
|
contain only affected counts and at most 25 one-based row numbers; they never
|
|
copy field values. Promotion copies this validation object into the immutable
|
|
materialization provenance and records the policy hash in the audit event.
|
|
|
|
Producer publication uses the same gate before any catalogue target,
|
|
materialization, or publication record is persisted. A rejected output has no
|
|
partial catalogue effect. Successful output materializations retain the exact
|
|
validation result, policy version/hash, and schema classification in
|
|
`provenance.publication_validation`; the publication record retains the same
|
|
evidence for operational inspection. Dataflow, Workflow, and Reporting can
|
|
therefore consume an immutable output reference without re-running a possibly
|
|
changed quality policy. Publication also emits a transactional
|
|
`datasource.publication.published` audit/platform event; an enabled Audit
|
|
module stores it in the durable outbox, while reduced installations deliver it
|
|
through Core after commit.
|
|
|
|
PostgreSQL deployments serialize publication attempts by tenant, producer, and
|
|
idempotency key with a transaction-scoped advisory lock before replay lookup.
|
|
This makes concurrent retries from separate API or worker nodes converge on the
|
|
same publication and materialization rather than relying on a late uniqueness
|
|
failure after output rows have already been persisted.
|
|
|
|
## Durable artifact publications
|
|
|
|
Outputs larger than the inline row and byte limits use an immutable artifact
|
|
reference. The reference pins its backend and locator together with SHA-256
|
|
checksum, schema, datasource fingerprint, row count, byte count, media type,
|
|
and optional resume checkpoint. Datasources persists that reference as the
|
|
materialization payload and asks the installed Core-contract artifact backend
|
|
to verify it before creating catalogue state. Reads remain bounded and are
|
|
re-authorized by Datasources before reaching the backend.
|
|
|
|
Schema rules are evaluated by Datasources. Content-level rules such as
|
|
uniqueness or range require producer evidence bound to the exact payload
|
|
checksum and current quality-policy hash, including all evaluated rule IDs.
|
|
Missing or explicitly deferred evidence produces a `review_required`
|
|
publication and an immutable, addressable materialization, but it does not
|
|
replace the Datasource's current state. Valid warnings produce
|
|
`published_with_warnings`; failed evidence blocks the publication without a
|
|
catalogue side effect. These terminal states are preserved for Dataflow and
|
|
Workflow handoffs instead of being collapsed into generic success.
|
|
|
|
Approval authority, approval expiry, and retention/deletion execution remain
|
|
separate work under `govoplan-datasources#2`. Until those contracts are added,
|
|
no JSON flag is treated as an approval and no stage is deleted automatically.
|