Datasource Quality Policy

Datasource quality policy is a deterministic JSON contract stored in governance.quality_policy. A tabular stage or producer publication inherits the current target Datasource policy unless it supplies its own governed definition. Stage validation runs before the stage is stored, but a failed stage remains available for inspection and correction through a new stage.

Contract

{
  "version": "monthly-import-v1",
  "rules": [
    {"id": "non-empty", "type": "row_count", "minimum": 1},
    {"id": "columns", "type": "required_fields", "fields": ["id", "status"]},
    {"id": "id-shape", "type": "field", "field": "id", "data_type": "integer", "nullable": false},
    {"id": "id-present", "type": "not_null", "fields": ["id"]},
    {"id": "id-unique", "type": "unique", "fields": ["id"]},
    {"id": "amount-range", "type": "range", "field": "amount", "minimum": 0},
    {"id": "known-status", "type": "referential", "field": "status", "allowed_values": ["new", "closed"]}
  ],
  "schema_policy": {
    "field_added_required": "warning",
    "field_removed": "breaking"
  }
}

Every rule has a stable id and may set severity to error or warning. Errors block promotion; warnings require review but leave the stage ready. The supported rules are:

row_count: optional non-negative minimum and maximum;
required_fields: fields that must exist in the detected schema;
field: expected data_type and/or nullable contract for one field;
not_null: one field or a fields list that must contain no nulls;
unique: one field or a composite fields list, with optional ignore_nulls;
range: numeric minimum and/or maximum, with optional allow_null;
referential: a field and a bounded allowed_values set, with optional allow_null.

The bounded referential rule is intentionally local and reproducible. A future cross-Datasource reference rule must freeze the referenced materialization and perform an independent read-authorization check; it must not silently read the current state of another protected Datasource.

Use embedded values only for non-sensitive code sets. Quality policy is catalogue governance metadata and can be visible to actors who are not allowed to read protected rows.

Schema Classification

When a stage targets an existing Datasource, fields are compared by stable name. Defaults are:

Change	Default
nullable field added	compatible
required field added	warning
field removed	breaking
integer widened to number, or unknown resolved	compatible
other type change	breaking
nullability relaxed	breaking
nullability tightened	compatible
existing field order changed	warning

Each key may be overridden in schema_policy with compatible, warning, or breaking. The highest resulting classification is the stage classification. A breaking classification blocks promotion.

Evidence And Privacy

Validation records the policy contract version, a SHA-256 hash of the complete policy, rule counts, diagnostics, and the complete schema diff. Row diagnostics contain only affected counts and at most 25 one-based row numbers; they never copy field values. Promotion copies this validation object into the immutable materialization provenance and records the policy hash in the audit event.

Producer publication uses the same gate before any catalogue target, materialization, or publication record is persisted. A rejected output has no partial catalogue effect. Successful output materializations retain the exact validation result, policy version/hash, and schema classification in provenance.publication_validation; the publication record retains the same evidence for operational inspection. Dataflow, Workflow, and Reporting can therefore consume an immutable output reference without re-running a possibly changed quality policy. Publication also emits a transactional datasource.publication.published audit/platform event; an enabled Audit module stores it in the durable outbox, while reduced installations deliver it through Core after commit.

PostgreSQL deployments serialize publication attempts by tenant, producer, and idempotency key with a transaction-scoped advisory lock before replay lookup. This makes concurrent retries from separate API or worker nodes converge on the same publication and materialization rather than relying on a late uniqueness failure after output rows have already been persisted.

Durable artifact publications

Outputs larger than the inline row and byte limits use an immutable artifact reference. The reference pins its backend and locator together with SHA-256 checksum, schema, datasource fingerprint, row count, byte count, media type, and optional resume checkpoint. Datasources persists that reference as the materialization payload and asks the installed Core-contract artifact backend to verify it before creating catalogue state. Reads remain bounded and are re-authorized by Datasources before reaching the backend.

Schema rules are evaluated by Datasources. Content-level rules such as uniqueness or range require producer evidence bound to the exact payload checksum and current quality-policy hash, including all evaluated rule IDs. Missing or explicitly deferred evidence produces a review_required publication and an immutable, addressable materialization, but it does not replace the Datasource's current state. Valid warnings produce published_with_warnings; failed evidence blocks the publication without a catalogue side effect. These terminal states are preserved for Dataflow and Workflow handoffs instead of being collapsed into generic success.

Promotion approval

Approval is a separate deterministic contract in governance.approval_policy:

{
  "version": "monthly-promotion-v2",
  "required": true,
  "required_approvals": 2,
  "separation_of_duties": true,
  "expires_after_hours": 72,
  "policy_ref": "policy:monthly-register-promotion"
}

A valid stage enters awaiting_approval instead of ready. Each decision is bound to the stage fingerprint, quality-policy hash, normalized approval-policy hash, actor authority, reason, and expiry. Actors are distinct and the stage creator cannot approve when separation of duties is enabled. The exact quorum snapshot and hash-chained evidence are copied into materialization provenance. A changed target policy or staged subject requires a new stage; request JSON cannot claim that an approval occurred.

Cached origins use POST /datasources/{id}/refresh/stage when approval is required. Direct refresh then fails closed, so connector content cannot become current before the staged validation and approval quorum succeed.

Retention

Retention is independently configured in governance.retention_policy:

{
  "version": "register-retention-v3",
  "enabled": true,
  "stage_days": 30,
  "materialization_days": 365,
  "frozen_evidence_days": 3650,
  "policy_ref": "policy:register-retention"
}

Durations are optional; omitting a class retains it indefinitely. Empty or disabled local policy never deletes content. An administrator first requests a read-only plan. The plan includes all due targets, policy versions and hashes, eligibility dates, blockers, and one hash over the complete preview. Applying retention accepts only targets from the unchanged plan.

Pending approvals, current materializations, legal holds, and materializations referenced by producer publications are blocked. Eligible stages are deleted. Eligible materialization payload rows are purged while the revision retains its schema, provenance, row/byte counts, payload checksum, disposition metadata, and immutable hash-chained lifecycle evidence. Frozen evidence is eligible only when frozen_evidence_days is explicitly configured. No hidden scheduler or arbitrary Policy/Access flag performs deletion.

7.8 KiB Raw Blame History