175 lines
7.8 KiB
Markdown
175 lines
7.8 KiB
Markdown
# Datasource Quality Policy
|
|
|
|
Datasource quality policy is a deterministic JSON contract stored in
|
|
`governance.quality_policy`. A tabular stage or producer publication inherits
|
|
the current target Datasource policy unless it supplies its own governed
|
|
definition. Stage validation runs before the stage is stored, but a failed
|
|
stage remains available for inspection and correction through a new stage.
|
|
|
|
## Contract
|
|
|
|
```json
|
|
{
|
|
"version": "monthly-import-v1",
|
|
"rules": [
|
|
{"id": "non-empty", "type": "row_count", "minimum": 1},
|
|
{"id": "columns", "type": "required_fields", "fields": ["id", "status"]},
|
|
{"id": "id-shape", "type": "field", "field": "id", "data_type": "integer", "nullable": false},
|
|
{"id": "id-present", "type": "not_null", "fields": ["id"]},
|
|
{"id": "id-unique", "type": "unique", "fields": ["id"]},
|
|
{"id": "amount-range", "type": "range", "field": "amount", "minimum": 0},
|
|
{"id": "known-status", "type": "referential", "field": "status", "allowed_values": ["new", "closed"]}
|
|
],
|
|
"schema_policy": {
|
|
"field_added_required": "warning",
|
|
"field_removed": "breaking"
|
|
}
|
|
}
|
|
```
|
|
|
|
Every rule has a stable `id` and may set `severity` to `error` or `warning`.
|
|
Errors block promotion; warnings require review but leave the stage ready. The
|
|
supported rules are:
|
|
|
|
- `row_count`: optional non-negative `minimum` and `maximum`;
|
|
- `required_fields`: fields that must exist in the detected schema;
|
|
- `field`: expected `data_type` and/or `nullable` contract for one field;
|
|
- `not_null`: one `field` or a `fields` list that must contain no nulls;
|
|
- `unique`: one `field` or a composite `fields` list, with optional
|
|
`ignore_nulls`;
|
|
- `range`: numeric `minimum` and/or `maximum`, with optional `allow_null`;
|
|
- `referential`: a field and a bounded `allowed_values` set, with optional
|
|
`allow_null`.
|
|
|
|
The bounded referential rule is intentionally local and reproducible. A future
|
|
cross-Datasource reference rule must freeze the referenced materialization and
|
|
perform an independent read-authorization check; it must not silently read the
|
|
current state of another protected Datasource.
|
|
|
|
Use embedded values only for non-sensitive code sets. Quality policy is
|
|
catalogue governance metadata and can be visible to actors who are not allowed
|
|
to read protected rows.
|
|
|
|
## Schema Classification
|
|
|
|
When a stage targets an existing Datasource, fields are compared by stable
|
|
name. Defaults are:
|
|
|
|
| Change | Default |
|
|
| --- | --- |
|
|
| nullable field added | compatible |
|
|
| required field added | warning |
|
|
| field removed | breaking |
|
|
| integer widened to number, or unknown resolved | compatible |
|
|
| other type change | breaking |
|
|
| nullability relaxed | breaking |
|
|
| nullability tightened | compatible |
|
|
| existing field order changed | warning |
|
|
|
|
Each key may be overridden in `schema_policy` with `compatible`, `warning`, or
|
|
`breaking`. The highest resulting classification is the stage classification.
|
|
A breaking classification blocks promotion.
|
|
|
|
## Evidence And Privacy
|
|
|
|
Validation records the policy contract version, a SHA-256 hash of the complete
|
|
policy, rule counts, diagnostics, and the complete schema diff. Row diagnostics
|
|
contain only affected counts and at most 25 one-based row numbers; they never
|
|
copy field values. Promotion copies this validation object into the immutable
|
|
materialization provenance and records the policy hash in the audit event.
|
|
|
|
Producer publication uses the same gate before any catalogue target,
|
|
materialization, or publication record is persisted. A rejected output has no
|
|
partial catalogue effect. Successful output materializations retain the exact
|
|
validation result, policy version/hash, and schema classification in
|
|
`provenance.publication_validation`; the publication record retains the same
|
|
evidence for operational inspection. Dataflow, Workflow, and Reporting can
|
|
therefore consume an immutable output reference without re-running a possibly
|
|
changed quality policy. Publication also emits a transactional
|
|
`datasource.publication.published` audit/platform event; an enabled Audit
|
|
module stores it in the durable outbox, while reduced installations deliver it
|
|
through Core after commit.
|
|
|
|
PostgreSQL deployments serialize publication attempts by tenant, producer, and
|
|
idempotency key with a transaction-scoped advisory lock before replay lookup.
|
|
This makes concurrent retries from separate API or worker nodes converge on the
|
|
same publication and materialization rather than relying on a late uniqueness
|
|
failure after output rows have already been persisted.
|
|
|
|
## Durable artifact publications
|
|
|
|
Outputs larger than the inline row and byte limits use an immutable artifact
|
|
reference. The reference pins its backend and locator together with SHA-256
|
|
checksum, schema, datasource fingerprint, row count, byte count, media type,
|
|
and optional resume checkpoint. Datasources persists that reference as the
|
|
materialization payload and asks the installed Core-contract artifact backend
|
|
to verify it before creating catalogue state. Reads remain bounded and are
|
|
re-authorized by Datasources before reaching the backend.
|
|
|
|
Schema rules are evaluated by Datasources. Content-level rules such as
|
|
uniqueness or range require producer evidence bound to the exact payload
|
|
checksum and current quality-policy hash, including all evaluated rule IDs.
|
|
Missing or explicitly deferred evidence produces a `review_required`
|
|
publication and an immutable, addressable materialization, but it does not
|
|
replace the Datasource's current state. Valid warnings produce
|
|
`published_with_warnings`; failed evidence blocks the publication without a
|
|
catalogue side effect. These terminal states are preserved for Dataflow and
|
|
Workflow handoffs instead of being collapsed into generic success.
|
|
|
|
## Promotion approval
|
|
|
|
Approval is a separate deterministic contract in
|
|
`governance.approval_policy`:
|
|
|
|
```json
|
|
{
|
|
"version": "monthly-promotion-v2",
|
|
"required": true,
|
|
"required_approvals": 2,
|
|
"separation_of_duties": true,
|
|
"expires_after_hours": 72,
|
|
"policy_ref": "policy:monthly-register-promotion"
|
|
}
|
|
```
|
|
|
|
A valid stage enters `awaiting_approval` instead of `ready`. Each decision is
|
|
bound to the stage fingerprint, quality-policy hash, normalized approval-policy
|
|
hash, actor authority, reason, and expiry. Actors are distinct and the stage
|
|
creator cannot approve when separation of duties is enabled. The exact quorum
|
|
snapshot and hash-chained evidence are copied into materialization provenance.
|
|
A changed target policy or staged subject requires a new stage; request JSON
|
|
cannot claim that an approval occurred.
|
|
|
|
Cached origins use `POST /datasources/{id}/refresh/stage` when approval is
|
|
required. Direct refresh then fails closed, so connector content cannot become
|
|
current before the staged validation and approval quorum succeed.
|
|
|
|
## Retention
|
|
|
|
Retention is independently configured in `governance.retention_policy`:
|
|
|
|
```json
|
|
{
|
|
"version": "register-retention-v3",
|
|
"enabled": true,
|
|
"stage_days": 30,
|
|
"materialization_days": 365,
|
|
"frozen_evidence_days": 3650,
|
|
"policy_ref": "policy:register-retention"
|
|
}
|
|
```
|
|
|
|
Durations are optional; omitting a class retains it indefinitely. Empty or
|
|
disabled local policy never deletes content. An administrator first requests a
|
|
read-only plan. The plan includes all due targets, policy versions and hashes,
|
|
eligibility dates, blockers, and one hash over the complete preview. Applying
|
|
retention accepts only targets from the unchanged plan.
|
|
|
|
Pending approvals, current materializations, legal holds, and materializations
|
|
referenced by producer publications are blocked. Eligible stages are deleted.
|
|
Eligible materialization payload rows are purged while the revision retains its
|
|
schema, provenance, row/byte counts, payload checksum, disposition metadata,
|
|
and immutable hash-chained lifecycle evidence. Frozen evidence is eligible only
|
|
when `frozen_evidence_days` is explicitly configured. No hidden scheduler or
|
|
arbitrary Policy/Access flag performs deletion.
|