Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7a7654cc0f |
@@ -38,6 +38,14 @@ promotion preserves the exact policy hash and validation result in immutable
|
|||||||
materialization provenance. The supported contract is documented in
|
materialization provenance. The supported contract is documented in
|
||||||
[docs/QUALITY_POLICY.md](docs/QUALITY_POLICY.md).
|
[docs/QUALITY_POLICY.md](docs/QUALITY_POLICY.md).
|
||||||
|
|
||||||
|
The same lifecycle contract can require an attributable, separated approval
|
||||||
|
quorum before a stage or cached refresh becomes current. Versioned retention
|
||||||
|
rules cover transient stages, ordinary materializations, and frozen evidence.
|
||||||
|
Retention remains an explicit preview-and-apply operation: current revisions,
|
||||||
|
legal holds, pending approvals, and publication evidence are blocked, while
|
||||||
|
every decision, promotion, and disposition is preserved in hash-chained local
|
||||||
|
evidence even when the optional Audit, Access, or Policy modules are absent.
|
||||||
|
|
||||||
The contracts already model database, HTTP/REST, directory, file, feed,
|
The contracts already model database, HTTP/REST, directory, file, feed,
|
||||||
document, binary, directory, and stream sources so providers can be added
|
document, binary, directory, and stream sources so providers can be added
|
||||||
without changing consumers. Storage modules contribute artifact backends
|
without changing consumers. Storage modules contribute artifact backends
|
||||||
@@ -49,7 +57,8 @@ See [docs/CONCEPT.md](docs/CONCEPT.md) for ownership and lifecycle details.
|
|||||||
## Data-subject requests
|
## Data-subject requests
|
||||||
|
|
||||||
Datasources publishes `privacy.dsar.datasources` for exact catalogue,
|
Datasources publishes `privacy.dsar.datasources` for exact catalogue,
|
||||||
governance-reference, materialization, payload, stage, and publication
|
governance-reference, materialization, payload, stage, publication, and
|
||||||
|
lifecycle-evidence
|
||||||
references and for minimized operator attribution. It never exports connector
|
references and for minimized operator attribution. It never exports connector
|
||||||
references, locators, credentials, arbitrary rows, schemas, validation
|
references, locators, credentials, arbitrary rows, schemas, validation
|
||||||
samples, metadata, provenance bodies, checkpoints, replay material, or hashes.
|
samples, metadata, provenance bodies, checkpoints, replay material, or hashes.
|
||||||
@@ -57,6 +66,7 @@ The module does not guess subject identity by scanning schema-dependent tabular
|
|||||||
payloads; the authoritative source module locates and corrects those facts.
|
payloads; the authoritative source module locates and corrects those facts.
|
||||||
|
|
||||||
Unpromoted stages and unreferenced payloads can be deleted idempotently.
|
Unpromoted stages and unreferenced payloads can be deleted idempotently.
|
||||||
Published or referenced state, immutable materializations, governance evidence,
|
Published or referenced state, immutable materializations, lifecycle evidence,
|
||||||
|
governance evidence,
|
||||||
holds, and operator attribution require data-steward review. Dataflow and
|
holds, and operator attribution require data-steward review. Dataflow and
|
||||||
Reporting derivatives must be refreshed after the source correction.
|
Reporting derivatives must be refreshed after the source correction.
|
||||||
|
|||||||
@@ -88,6 +88,21 @@ The first slice stores bounded tabular JSON/CSV stages. Future providers may
|
|||||||
stage file references, object-store blobs, directory snapshots, or streaming
|
stage file references, object-store blobs, directory snapshots, or streaming
|
||||||
checkpoints through the same lifecycle contract.
|
checkpoints through the same lifecycle contract.
|
||||||
|
|
||||||
|
Valid stages may be subject to a versioned local approval policy. Required
|
||||||
|
quorums keep the stage non-consumable, enforce distinct attributable actors and
|
||||||
|
optional creator/approver separation, and bind every decision to exact policy
|
||||||
|
and subject hashes. Cached refreshes use this same staging path when approval is
|
||||||
|
required. An installation without separate Access or Policy modules retains
|
||||||
|
safe local scope and policy defaults; optional modules do not manufacture an
|
||||||
|
approval from configuration metadata.
|
||||||
|
|
||||||
|
Retention policy separately defines durations for stages, ordinary
|
||||||
|
materializations, and frozen evidence. Administrators receive a deterministic
|
||||||
|
plan before applying any disposition. Holds, current state, pending decisions,
|
||||||
|
and publication evidence are blockers. Payload disposal preserves a minimized
|
||||||
|
materialization tombstone and hash-chained lifecycle evidence, so chronological
|
||||||
|
authority and policy proof survive the configured content-retention boundary.
|
||||||
|
|
||||||
## Consumer Contract
|
## Consumer Contract
|
||||||
|
|
||||||
Consumers request:
|
Consumers request:
|
||||||
|
|||||||
+56
-3
@@ -116,6 +116,59 @@ replace the Datasource's current state. Valid warnings produce
|
|||||||
catalogue side effect. These terminal states are preserved for Dataflow and
|
catalogue side effect. These terminal states are preserved for Dataflow and
|
||||||
Workflow handoffs instead of being collapsed into generic success.
|
Workflow handoffs instead of being collapsed into generic success.
|
||||||
|
|
||||||
Approval authority, approval expiry, and retention/deletion execution remain
|
## Promotion approval
|
||||||
separate work under `govoplan-datasources#2`. Until those contracts are added,
|
|
||||||
no JSON flag is treated as an approval and no stage is deleted automatically.
|
Approval is a separate deterministic contract in
|
||||||
|
`governance.approval_policy`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"version": "monthly-promotion-v2",
|
||||||
|
"required": true,
|
||||||
|
"required_approvals": 2,
|
||||||
|
"separation_of_duties": true,
|
||||||
|
"expires_after_hours": 72,
|
||||||
|
"policy_ref": "policy:monthly-register-promotion"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
A valid stage enters `awaiting_approval` instead of `ready`. Each decision is
|
||||||
|
bound to the stage fingerprint, quality-policy hash, normalized approval-policy
|
||||||
|
hash, actor authority, reason, and expiry. Actors are distinct and the stage
|
||||||
|
creator cannot approve when separation of duties is enabled. The exact quorum
|
||||||
|
snapshot and hash-chained evidence are copied into materialization provenance.
|
||||||
|
A changed target policy or staged subject requires a new stage; request JSON
|
||||||
|
cannot claim that an approval occurred.
|
||||||
|
|
||||||
|
Cached origins use `POST /datasources/{id}/refresh/stage` when approval is
|
||||||
|
required. Direct refresh then fails closed, so connector content cannot become
|
||||||
|
current before the staged validation and approval quorum succeed.
|
||||||
|
|
||||||
|
## Retention
|
||||||
|
|
||||||
|
Retention is independently configured in `governance.retention_policy`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"version": "register-retention-v3",
|
||||||
|
"enabled": true,
|
||||||
|
"stage_days": 30,
|
||||||
|
"materialization_days": 365,
|
||||||
|
"frozen_evidence_days": 3650,
|
||||||
|
"policy_ref": "policy:register-retention"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Durations are optional; omitting a class retains it indefinitely. Empty or
|
||||||
|
disabled local policy never deletes content. An administrator first requests a
|
||||||
|
read-only plan. The plan includes all due targets, policy versions and hashes,
|
||||||
|
eligibility dates, blockers, and one hash over the complete preview. Applying
|
||||||
|
retention accepts only targets from the unchanged plan.
|
||||||
|
|
||||||
|
Pending approvals, current materializations, legal holds, and materializations
|
||||||
|
referenced by producer publications are blocked. Eligible stages are deleted.
|
||||||
|
Eligible materialization payload rows are purged while the revision retains its
|
||||||
|
schema, provenance, row/byte counts, payload checksum, disposition metadata,
|
||||||
|
and immutable hash-chained lifecycle evidence. Frozen evidence is eligible only
|
||||||
|
when `frozen_evidence_days` is explicitly configured. No hidden scheduler or
|
||||||
|
arbitrary Policy/Access flag performs deletion.
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "govoplan-datasources"
|
name = "govoplan-datasources"
|
||||||
version = "0.1.20"
|
version = "0.1.21"
|
||||||
description = "Governed datasource catalogue, staging, and materialization lifecycle for GovOPlaN."
|
description = "Governed datasource catalogue, staging, and materialization lifecycle for GovOPlaN."
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.12"
|
requires-python = ">=3.12"
|
||||||
|
|||||||
@@ -1,3 +1,3 @@
|
|||||||
"""GovOPlaN Datasources module."""
|
"""GovOPlaN Datasources module."""
|
||||||
|
|
||||||
__version__ = "0.1.20"
|
__version__ = "0.1.21"
|
||||||
|
|||||||
@@ -126,6 +126,12 @@ class DatasourceRecord(Base, TimestampMixin):
|
|||||||
quality_policy: Mapped[dict[str, Any]] = mapped_column(
|
quality_policy: Mapped[dict[str, Any]] = mapped_column(
|
||||||
JSON, default=dict, nullable=False
|
JSON, default=dict, nullable=False
|
||||||
)
|
)
|
||||||
|
approval_policy: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
|
retention_policy: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
known_limits: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
|
known_limits: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
|
||||||
correction_procedure_ref: Mapped[str | None] = mapped_column(
|
correction_procedure_ref: Mapped[str | None] = mapped_column(
|
||||||
String(500), nullable=True
|
String(500), nullable=True
|
||||||
@@ -266,6 +272,15 @@ class DatasourceMaterializationRecord(Base, TimestampMixin):
|
|||||||
default=dict,
|
default=dict,
|
||||||
nullable=False,
|
nullable=False,
|
||||||
)
|
)
|
||||||
|
disposed_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True, index=True
|
||||||
|
)
|
||||||
|
disposition_: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
"disposition",
|
||||||
|
JSON,
|
||||||
|
default=dict,
|
||||||
|
nullable=False,
|
||||||
|
)
|
||||||
created_by: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
created_by: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
||||||
|
|
||||||
datasource: Mapped[DatasourceRecord] = relationship(back_populates="materializations")
|
datasource: Mapped[DatasourceRecord] = relationship(back_populates="materializations")
|
||||||
@@ -417,6 +432,12 @@ class DatasourceStageRecord(Base, TimestampMixin):
|
|||||||
default=dict,
|
default=dict,
|
||||||
nullable=False,
|
nullable=False,
|
||||||
)
|
)
|
||||||
|
approval_: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
"approval",
|
||||||
|
JSON,
|
||||||
|
default=dict,
|
||||||
|
nullable=False,
|
||||||
|
)
|
||||||
promoted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
promoted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||||
promoted_materialization_id: Mapped[str | None] = mapped_column(
|
promoted_materialization_id: Mapped[str | None] = mapped_column(
|
||||||
String(36),
|
String(36),
|
||||||
@@ -483,7 +504,49 @@ class DatasourcePublicationRecord(Base, TimestampMixin):
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceLifecycleEvidenceRecord(Base):
|
||||||
|
__tablename__ = "datasource_lifecycle_evidence"
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"event_hash",
|
||||||
|
name="uq_datasource_lifecycle_evidence_hash",
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"ix_datasource_lifecycle_evidence_subject",
|
||||||
|
"tenant_id",
|
||||||
|
"subject_ref",
|
||||||
|
"occurred_at",
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"ix_datasource_lifecycle_evidence_event",
|
||||||
|
"tenant_id",
|
||||||
|
"event_type",
|
||||||
|
"occurred_at",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
|
subject_ref: Mapped[str] = mapped_column(String(160), nullable=False, index=True)
|
||||||
|
event_type: Mapped[str] = mapped_column(String(80), nullable=False, index=True)
|
||||||
|
occurred_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||||
|
actor_ref: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
||||||
|
policy_version: Mapped[str | None] = mapped_column(String(120), nullable=True)
|
||||||
|
policy_hash: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||||
|
subject_digest: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||||
|
previous_event_hash: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||||
|
event_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||||
|
details_: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
"details",
|
||||||
|
JSON,
|
||||||
|
default=dict,
|
||||||
|
nullable=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
|
"DatasourceLifecycleEvidenceRecord",
|
||||||
"DatasourceMaterializationRecord",
|
"DatasourceMaterializationRecord",
|
||||||
"DatasourcePayloadRecord",
|
"DatasourcePayloadRecord",
|
||||||
"DatasourcePayloadRowRecord",
|
"DatasourcePayloadRowRecord",
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ from govoplan_core.core.dsar import (
|
|||||||
)
|
)
|
||||||
from govoplan_datasources.backend.db.models import (
|
from govoplan_datasources.backend.db.models import (
|
||||||
DatasourceGovernanceReferenceRecord,
|
DatasourceGovernanceReferenceRecord,
|
||||||
|
DatasourceLifecycleEvidenceRecord,
|
||||||
DatasourceMaterializationRecord,
|
DatasourceMaterializationRecord,
|
||||||
DatasourcePayloadRecord,
|
DatasourcePayloadRecord,
|
||||||
DatasourcePublicationRecord,
|
DatasourcePublicationRecord,
|
||||||
@@ -35,6 +36,7 @@ _DIRECT_ALIASES = {
|
|||||||
"payload_id": ("datasources.payload",),
|
"payload_id": ("datasources.payload",),
|
||||||
"stage_id": ("datasources.stage",),
|
"stage_id": ("datasources.stage",),
|
||||||
"publication_id": ("datasources.publication",),
|
"publication_id": ("datasources.publication",),
|
||||||
|
"lifecycle_evidence_id": ("datasources.lifecycle_evidence",),
|
||||||
}
|
}
|
||||||
_RESOURCE_MODELS = {
|
_RESOURCE_MODELS = {
|
||||||
"datasource": DatasourceRecord,
|
"datasource": DatasourceRecord,
|
||||||
@@ -43,6 +45,7 @@ _RESOURCE_MODELS = {
|
|||||||
"datasource_payload": DatasourcePayloadRecord,
|
"datasource_payload": DatasourcePayloadRecord,
|
||||||
"datasource_stage": DatasourceStageRecord,
|
"datasource_stage": DatasourceStageRecord,
|
||||||
"datasource_publication": DatasourcePublicationRecord,
|
"datasource_publication": DatasourcePublicationRecord,
|
||||||
|
"datasource_lifecycle_evidence": DatasourceLifecycleEvidenceRecord,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -284,6 +287,10 @@ def _direct_matches(
|
|||||||
DatasourcePublicationRecord,
|
DatasourcePublicationRecord,
|
||||||
"datasource_publication",
|
"datasource_publication",
|
||||||
),
|
),
|
||||||
|
"lifecycle_evidence_id": (
|
||||||
|
DatasourceLifecycleEvidenceRecord,
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
),
|
||||||
}[selector]
|
}[selector]
|
||||||
row = _one(
|
row = _one(
|
||||||
session,
|
session,
|
||||||
@@ -421,6 +428,11 @@ def _canonical_matches(
|
|||||||
DatasourcePublicationRecord.created_by.in_(actor_ids),
|
DatasourcePublicationRecord.created_by.in_(actor_ids),
|
||||||
"datasource_publication",
|
"datasource_publication",
|
||||||
),
|
),
|
||||||
|
(
|
||||||
|
DatasourceLifecycleEvidenceRecord,
|
||||||
|
DatasourceLifecycleEvidenceRecord.actor_ref.in_(actor_ids),
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
),
|
||||||
)
|
)
|
||||||
matches: list[_Match] = []
|
matches: list[_Match] = []
|
||||||
for model, condition, resource_type in specs:
|
for model, condition, resource_type in specs:
|
||||||
@@ -472,6 +484,8 @@ def _direct_category(session: Session, resource_type: str, row: Any) -> str:
|
|||||||
)
|
)
|
||||||
if not referenced:
|
if not referenced:
|
||||||
return "unreferenced_datasource_payload"
|
return "unreferenced_datasource_payload"
|
||||||
|
if resource_type == "datasource_lifecycle_evidence":
|
||||||
|
return "datasource_operator_attribution"
|
||||||
return {
|
return {
|
||||||
"datasource": "datasource_configuration",
|
"datasource": "datasource_configuration",
|
||||||
"datasource_governance_reference": "datasource_governance_configuration",
|
"datasource_governance_reference": "datasource_governance_configuration",
|
||||||
@@ -479,6 +493,7 @@ def _direct_category(session: Session, resource_type: str, row: Any) -> str:
|
|||||||
"datasource_payload": "referenced_datasource_payload",
|
"datasource_payload": "referenced_datasource_payload",
|
||||||
"datasource_stage": "promoted_datasource_stage",
|
"datasource_stage": "promoted_datasource_stage",
|
||||||
"datasource_publication": "immutable_datasource_publication",
|
"datasource_publication": "immutable_datasource_publication",
|
||||||
|
"datasource_lifecycle_evidence": "datasource_operator_attribution",
|
||||||
}[resource_type]
|
}[resource_type]
|
||||||
|
|
||||||
|
|
||||||
@@ -498,6 +513,22 @@ def _root_datasource_ids(session: Session, match: _Match) -> set[str]:
|
|||||||
)
|
)
|
||||||
.all()
|
.all()
|
||||||
}
|
}
|
||||||
|
if match.resource_type == "datasource_lifecycle_evidence":
|
||||||
|
if row.subject_ref.startswith("datasource:"):
|
||||||
|
return {row.subject_ref.removeprefix("datasource:")}
|
||||||
|
if row.subject_ref.startswith("stage:"):
|
||||||
|
stage = session.get(
|
||||||
|
DatasourceStageRecord,
|
||||||
|
row.subject_ref.removeprefix("stage:"),
|
||||||
|
)
|
||||||
|
return {stage.target_datasource_id} if stage and stage.target_datasource_id else set()
|
||||||
|
if row.subject_ref.startswith("materialization:"):
|
||||||
|
materialization = session.get(
|
||||||
|
DatasourceMaterializationRecord,
|
||||||
|
row.subject_ref.removeprefix("materialization:"),
|
||||||
|
)
|
||||||
|
return {materialization.datasource_id} if materialization else set()
|
||||||
|
return set()
|
||||||
return {row.datasource_id}
|
return {row.datasource_id}
|
||||||
|
|
||||||
|
|
||||||
@@ -505,7 +536,7 @@ def _correlates(match: _Match, actor_ids: tuple[str, ...]) -> bool:
|
|||||||
row = match.row
|
row = match.row
|
||||||
return any(
|
return any(
|
||||||
str(getattr(row, field, "") or "") in actor_ids
|
str(getattr(row, field, "") or "") in actor_ids
|
||||||
for field in ("created_by", "updated_by")
|
for field in ("created_by", "updated_by", "actor_ref")
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -521,6 +552,7 @@ def _directly_targets(selectors: _Selectors, match: _Match) -> bool:
|
|||||||
"datasource_payload": ("payload_id", "id"),
|
"datasource_payload": ("payload_id", "id"),
|
||||||
"datasource_stage": ("stage_id", "id"),
|
"datasource_stage": ("stage_id", "id"),
|
||||||
"datasource_publication": ("publication_id", "id"),
|
"datasource_publication": ("publication_id", "id"),
|
||||||
|
"datasource_lifecycle_evidence": ("lifecycle_evidence_id", "id"),
|
||||||
}[match.resource_type]
|
}[match.resource_type]
|
||||||
value = _strip_prefix(selectors.direct.get(selector, ""))
|
value = _strip_prefix(selectors.direct.get(selector, ""))
|
||||||
if value == str(getattr(row, field)):
|
if value == str(getattr(row, field)):
|
||||||
@@ -540,6 +572,12 @@ def _root_datasource_ids_for_row(match: _Match) -> set[str]:
|
|||||||
return {row.target_datasource_id} if row.target_datasource_id else set()
|
return {row.target_datasource_id} if row.target_datasource_id else set()
|
||||||
if match.resource_type == "datasource_payload":
|
if match.resource_type == "datasource_payload":
|
||||||
return set()
|
return set()
|
||||||
|
if match.resource_type == "datasource_lifecycle_evidence":
|
||||||
|
return (
|
||||||
|
{row.subject_ref.removeprefix("datasource:")}
|
||||||
|
if row.subject_ref.startswith("datasource:")
|
||||||
|
else set()
|
||||||
|
)
|
||||||
return {row.datasource_id}
|
return {row.datasource_id}
|
||||||
|
|
||||||
|
|
||||||
@@ -605,6 +643,13 @@ def _record_data(resource_type: str, row: Any) -> dict[str, object]:
|
|||||||
"created_at": _iso(row.created_at),
|
"created_at": _iso(row.created_at),
|
||||||
"updated_at": _iso(row.updated_at),
|
"updated_at": _iso(row.updated_at),
|
||||||
}
|
}
|
||||||
|
if resource_type == "datasource_lifecycle_evidence":
|
||||||
|
return {
|
||||||
|
"subject_ref": row.subject_ref,
|
||||||
|
"event_type": row.event_type,
|
||||||
|
"policy_version": row.policy_version,
|
||||||
|
"occurred_at": _iso(row.occurred_at),
|
||||||
|
}
|
||||||
if resource_type == "datasource_governance_reference":
|
if resource_type == "datasource_governance_reference":
|
||||||
return {"relation": row.relation}
|
return {"relation": row.relation}
|
||||||
if resource_type == "datasource_materialization":
|
if resource_type == "datasource_materialization":
|
||||||
@@ -700,7 +745,13 @@ def _title(resource_type: str) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def _observed_at(row: Any) -> datetime | None:
|
def _observed_at(row: Any) -> datetime | None:
|
||||||
for field in ("promoted_at", "source_timestamp", "updated_at", "created_at"):
|
for field in (
|
||||||
|
"promoted_at",
|
||||||
|
"source_timestamp",
|
||||||
|
"occurred_at",
|
||||||
|
"updated_at",
|
||||||
|
"created_at",
|
||||||
|
):
|
||||||
value = getattr(row, field, None)
|
value = getattr(row, field, None)
|
||||||
if isinstance(value, datetime):
|
if isinstance(value, datetime):
|
||||||
return _aware(value)
|
return _aware(value)
|
||||||
|
|||||||
@@ -0,0 +1,718 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
|
||||||
|
from sqlalchemy import func, select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.datasources import DatasourceValidationError
|
||||||
|
from govoplan_core.db.base import utcnow
|
||||||
|
from govoplan_datasources.backend.db.models import (
|
||||||
|
DatasourceLifecycleEvidenceRecord,
|
||||||
|
DatasourceMaterializationRecord,
|
||||||
|
DatasourcePublicationRecord,
|
||||||
|
DatasourceRecord,
|
||||||
|
DatasourceStageRecord,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class RetentionCandidate:
|
||||||
|
ref: str
|
||||||
|
kind: str
|
||||||
|
datasource_ref: str | None
|
||||||
|
disposition: str
|
||||||
|
eligible_at: datetime
|
||||||
|
eligible: bool
|
||||||
|
blockers: tuple[str, ...]
|
||||||
|
policy_version: str
|
||||||
|
policy_hash: str
|
||||||
|
|
||||||
|
def to_dict(self) -> dict[str, object]:
|
||||||
|
return {
|
||||||
|
"ref": self.ref,
|
||||||
|
"kind": self.kind,
|
||||||
|
"datasource_ref": self.datasource_ref,
|
||||||
|
"disposition": self.disposition,
|
||||||
|
"eligible_at": _as_utc(self.eligible_at).isoformat(),
|
||||||
|
"eligible": self.eligible,
|
||||||
|
"blockers": list(self.blockers),
|
||||||
|
"policy_version": self.policy_version,
|
||||||
|
"policy_hash": self.policy_hash,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class RetentionPlan:
|
||||||
|
as_of: datetime
|
||||||
|
plan_hash: str
|
||||||
|
candidates: tuple[RetentionCandidate, ...]
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_hash(value: object) -> str:
|
||||||
|
encoded = json.dumps(
|
||||||
|
value,
|
||||||
|
sort_keys=True,
|
||||||
|
separators=(",", ":"),
|
||||||
|
ensure_ascii=True,
|
||||||
|
default=str,
|
||||||
|
)
|
||||||
|
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_approval_policy(value: Mapping[str, object] | None) -> dict[str, object]:
|
||||||
|
source = dict(value or {})
|
||||||
|
version = str(source.get("version") or "1").strip()
|
||||||
|
if not version:
|
||||||
|
raise DatasourceValidationError("Approval policy version is required.")
|
||||||
|
required = _boolean(source.get("required", False), label="required")
|
||||||
|
required_approvals = _bounded_integer(
|
||||||
|
source.get("required_approvals", 1),
|
||||||
|
label="required_approvals",
|
||||||
|
minimum=1,
|
||||||
|
maximum=5,
|
||||||
|
)
|
||||||
|
separation = _boolean(
|
||||||
|
source.get("separation_of_duties", True),
|
||||||
|
label="separation_of_duties",
|
||||||
|
)
|
||||||
|
expires_after_hours = source.get("expires_after_hours")
|
||||||
|
expiry = (
|
||||||
|
None
|
||||||
|
if expires_after_hours is None
|
||||||
|
else _bounded_integer(
|
||||||
|
expires_after_hours,
|
||||||
|
label="expires_after_hours",
|
||||||
|
minimum=1,
|
||||||
|
maximum=8_760,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
policy_ref = _optional_text(source.get("policy_ref"))
|
||||||
|
return {
|
||||||
|
"version": version,
|
||||||
|
"required": required,
|
||||||
|
"required_approvals": required_approvals,
|
||||||
|
"separation_of_duties": separation,
|
||||||
|
"expires_after_hours": expiry,
|
||||||
|
"policy_ref": policy_ref,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_retention_policy(value: Mapping[str, object] | None) -> dict[str, object]:
|
||||||
|
source = dict(value or {})
|
||||||
|
version = str(source.get("version") or "1").strip()
|
||||||
|
if not version:
|
||||||
|
raise DatasourceValidationError("Retention policy version is required.")
|
||||||
|
enabled = _boolean(source.get("enabled", False), label="enabled")
|
||||||
|
durations = {
|
||||||
|
key: _optional_duration(source.get(key), label=key)
|
||||||
|
for key in (
|
||||||
|
"stage_days",
|
||||||
|
"materialization_days",
|
||||||
|
"frozen_evidence_days",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
if enabled and all(value is None for value in durations.values()):
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"Enabled retention policy requires at least one retention duration."
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"version": version,
|
||||||
|
"enabled": enabled,
|
||||||
|
**durations,
|
||||||
|
"policy_ref": _optional_text(source.get("policy_ref")),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def initialize_stage_approval(
|
||||||
|
stage: DatasourceStageRecord,
|
||||||
|
*,
|
||||||
|
created_at: datetime | None = None,
|
||||||
|
) -> dict[str, object]:
|
||||||
|
policy = normalize_approval_policy(
|
||||||
|
_mapping(stage.governance_).get("approval_policy")
|
||||||
|
if isinstance(_mapping(stage.governance_).get("approval_policy"), Mapping)
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
now = _as_utc(created_at or utcnow())
|
||||||
|
expiry_hours = policy["expires_after_hours"]
|
||||||
|
expires_at = (
|
||||||
|
now + timedelta(hours=int(expiry_hours))
|
||||||
|
if isinstance(expiry_hours, int)
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
approval = {
|
||||||
|
"state": "pending" if policy["required"] else "not_required",
|
||||||
|
"policy": policy,
|
||||||
|
"policy_hash": canonical_hash(policy),
|
||||||
|
"subject_digest": stage_subject_digest(stage, policy=policy),
|
||||||
|
"required_approvals": policy["required_approvals"],
|
||||||
|
"approval_count": 0,
|
||||||
|
"expires_at": expires_at.isoformat() if expires_at else None,
|
||||||
|
"approvals": [],
|
||||||
|
}
|
||||||
|
stage.approval_ = approval
|
||||||
|
if stage.validation_.get("valid") is True:
|
||||||
|
stage.state = "awaiting_approval" if policy["required"] else "ready"
|
||||||
|
return approval
|
||||||
|
|
||||||
|
|
||||||
|
def decide_stage(
|
||||||
|
stage: DatasourceStageRecord,
|
||||||
|
*,
|
||||||
|
actor_ref: str,
|
||||||
|
actor_scopes: Sequence[str],
|
||||||
|
decision: str,
|
||||||
|
reason: str,
|
||||||
|
expected_policy_hash: str,
|
||||||
|
expected_subject_digest: str,
|
||||||
|
now: datetime | None = None,
|
||||||
|
) -> tuple[dict[str, object], bool]:
|
||||||
|
approval = dict(stage.approval_ or {})
|
||||||
|
if approval.get("policy_hash") != expected_policy_hash:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"The approval policy changed; reload the stage before deciding."
|
||||||
|
)
|
||||||
|
if approval.get("subject_digest") != expected_subject_digest:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"The staged content changed; reload the stage before deciding."
|
||||||
|
)
|
||||||
|
policy = normalize_approval_policy(_mapping(approval.get("policy")))
|
||||||
|
if not policy["required"]:
|
||||||
|
raise DatasourceValidationError("This stage does not require approval.")
|
||||||
|
if stage.state not in {"awaiting_approval", "ready"}:
|
||||||
|
raise DatasourceValidationError("This stage no longer accepts approval decisions.")
|
||||||
|
if policy["separation_of_duties"] and stage.created_by == actor_ref:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"The stage creator cannot approve this stage under separation of duties."
|
||||||
|
)
|
||||||
|
decided_at = _as_utc(now or utcnow())
|
||||||
|
expires_at = _optional_datetime(approval.get("expires_at"))
|
||||||
|
if expires_at is not None and decided_at > expires_at:
|
||||||
|
approval["state"] = "expired"
|
||||||
|
stage.approval_ = approval
|
||||||
|
stage.state = "awaiting_approval"
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"The approval window expired; create a new stage for a fresh decision."
|
||||||
|
)
|
||||||
|
decisions = [dict(item) for item in _mapping_sequence(approval.get("approvals"))]
|
||||||
|
existing = next(
|
||||||
|
(item for item in decisions if item.get("actor_ref") == actor_ref),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if existing is not None:
|
||||||
|
if existing.get("decision") == decision:
|
||||||
|
return approval, True
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"This approver already recorded a different decision."
|
||||||
|
)
|
||||||
|
cleaned_reason = reason.strip()
|
||||||
|
if not cleaned_reason:
|
||||||
|
raise DatasourceValidationError("An approval decision reason is required.")
|
||||||
|
decisions.append(
|
||||||
|
{
|
||||||
|
"actor_ref": actor_ref,
|
||||||
|
"decision": decision,
|
||||||
|
"reason": cleaned_reason,
|
||||||
|
"decided_at": decided_at.isoformat(),
|
||||||
|
"authority_scopes": sorted(set(actor_scopes)),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
approved = sum(item.get("decision") == "approve" for item in decisions)
|
||||||
|
approval.update(
|
||||||
|
{
|
||||||
|
"approvals": decisions,
|
||||||
|
"approval_count": approved,
|
||||||
|
"state": (
|
||||||
|
"rejected"
|
||||||
|
if decision == "reject"
|
||||||
|
else (
|
||||||
|
"approved"
|
||||||
|
if approved >= int(policy["required_approvals"])
|
||||||
|
else "pending"
|
||||||
|
)
|
||||||
|
),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
stage.approval_ = approval
|
||||||
|
stage.state = "rejected" if decision == "reject" else (
|
||||||
|
"ready"
|
||||||
|
if approval["state"] == "approved"
|
||||||
|
else "awaiting_approval"
|
||||||
|
)
|
||||||
|
return approval, False
|
||||||
|
|
||||||
|
|
||||||
|
def ensure_stage_approval_current(
|
||||||
|
stage: DatasourceStageRecord,
|
||||||
|
*,
|
||||||
|
current_policy: Mapping[str, object] | None,
|
||||||
|
) -> None:
|
||||||
|
approval = _mapping(stage.approval_)
|
||||||
|
stage_policy = normalize_approval_policy(_mapping(approval.get("policy")))
|
||||||
|
effective = normalize_approval_policy(current_policy)
|
||||||
|
if canonical_hash(effective) != approval.get("policy_hash"):
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"The approval policy changed after staging; create a new stage."
|
||||||
|
)
|
||||||
|
if stage_policy["required"] and approval.get("state") != "approved":
|
||||||
|
raise DatasourceValidationError("The stage has not reached its approval quorum.")
|
||||||
|
if approval.get("subject_digest") != stage_subject_digest(stage, policy=stage_policy):
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"The staged evidence changed after approval; create a new stage."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def stage_subject_digest(
|
||||||
|
stage: DatasourceStageRecord,
|
||||||
|
*,
|
||||||
|
policy: Mapping[str, object] | None = None,
|
||||||
|
) -> str:
|
||||||
|
return canonical_hash(
|
||||||
|
{
|
||||||
|
"stage_ref": _stage_ref(stage.id),
|
||||||
|
"target_datasource_ref": (
|
||||||
|
_datasource_ref(stage.target_datasource_id)
|
||||||
|
if stage.target_datasource_id
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
"source_name": stage.source_name,
|
||||||
|
"mode": stage.mode,
|
||||||
|
"shape": stage.shape,
|
||||||
|
"fingerprint": stage.fingerprint,
|
||||||
|
"validation_policy_hash": stage.validation_.get("policy_hash"),
|
||||||
|
"approval_policy": dict(policy or {}),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def record_lifecycle_evidence(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
subject_ref: str,
|
||||||
|
event_type: str,
|
||||||
|
actor_ref: str | None,
|
||||||
|
subject_digest: str,
|
||||||
|
details: Mapping[str, object],
|
||||||
|
policy_version: str | None = None,
|
||||||
|
policy_hash: str | None = None,
|
||||||
|
occurred_at: datetime | None = None,
|
||||||
|
) -> DatasourceLifecycleEvidenceRecord:
|
||||||
|
happened = _as_utc(occurred_at or utcnow())
|
||||||
|
previous_hash = session.scalar(
|
||||||
|
select(DatasourceLifecycleEvidenceRecord.event_hash)
|
||||||
|
.where(
|
||||||
|
DatasourceLifecycleEvidenceRecord.tenant_id == tenant_id,
|
||||||
|
DatasourceLifecycleEvidenceRecord.subject_ref == subject_ref,
|
||||||
|
)
|
||||||
|
.order_by(
|
||||||
|
DatasourceLifecycleEvidenceRecord.occurred_at.desc(),
|
||||||
|
DatasourceLifecycleEvidenceRecord.id.desc(),
|
||||||
|
)
|
||||||
|
.limit(1)
|
||||||
|
)
|
||||||
|
payload = {
|
||||||
|
"tenant_id": tenant_id,
|
||||||
|
"subject_ref": subject_ref,
|
||||||
|
"event_type": event_type,
|
||||||
|
"occurred_at": happened.isoformat(),
|
||||||
|
"actor_ref": actor_ref,
|
||||||
|
"policy_version": policy_version,
|
||||||
|
"policy_hash": policy_hash,
|
||||||
|
"subject_digest": subject_digest,
|
||||||
|
"previous_event_hash": previous_hash,
|
||||||
|
"details": dict(details),
|
||||||
|
}
|
||||||
|
item = DatasourceLifecycleEvidenceRecord(
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
subject_ref=subject_ref,
|
||||||
|
event_type=event_type,
|
||||||
|
occurred_at=happened,
|
||||||
|
actor_ref=actor_ref,
|
||||||
|
policy_version=policy_version,
|
||||||
|
policy_hash=policy_hash,
|
||||||
|
subject_digest=subject_digest,
|
||||||
|
previous_event_hash=previous_hash,
|
||||||
|
event_hash=canonical_hash(payload),
|
||||||
|
details_=dict(details),
|
||||||
|
)
|
||||||
|
session.add(item)
|
||||||
|
session.flush()
|
||||||
|
return item
|
||||||
|
|
||||||
|
|
||||||
|
def list_lifecycle_evidence(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
subject_ref: str | None = None,
|
||||||
|
limit: int = 200,
|
||||||
|
) -> tuple[DatasourceLifecycleEvidenceRecord, ...]:
|
||||||
|
statement = select(DatasourceLifecycleEvidenceRecord).where(
|
||||||
|
DatasourceLifecycleEvidenceRecord.tenant_id == tenant_id
|
||||||
|
)
|
||||||
|
if subject_ref:
|
||||||
|
statement = statement.where(
|
||||||
|
DatasourceLifecycleEvidenceRecord.subject_ref == subject_ref
|
||||||
|
)
|
||||||
|
statement = statement.order_by(
|
||||||
|
DatasourceLifecycleEvidenceRecord.occurred_at.desc(),
|
||||||
|
DatasourceLifecycleEvidenceRecord.id.desc(),
|
||||||
|
).limit(max(1, min(limit, 500)))
|
||||||
|
return tuple(session.scalars(statement))
|
||||||
|
|
||||||
|
|
||||||
|
def build_retention_plan(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
as_of: datetime,
|
||||||
|
) -> RetentionPlan:
|
||||||
|
effective_at = _as_utc(as_of)
|
||||||
|
candidates: list[RetentionCandidate] = []
|
||||||
|
stages = session.scalars(
|
||||||
|
select(DatasourceStageRecord).where(
|
||||||
|
DatasourceStageRecord.tenant_id == tenant_id
|
||||||
|
)
|
||||||
|
)
|
||||||
|
for stage in stages:
|
||||||
|
policy = normalize_retention_policy(
|
||||||
|
_mapping(_mapping(stage.governance_).get("retention_policy"))
|
||||||
|
)
|
||||||
|
duration = policy.get("stage_days")
|
||||||
|
if not policy["enabled"] or not isinstance(duration, int):
|
||||||
|
continue
|
||||||
|
eligible_at = _as_utc(stage.created_at) + timedelta(days=duration)
|
||||||
|
if eligible_at > effective_at:
|
||||||
|
continue
|
||||||
|
blockers = (
|
||||||
|
("pending_approval",)
|
||||||
|
if stage.state == "awaiting_approval"
|
||||||
|
else ()
|
||||||
|
)
|
||||||
|
candidates.append(
|
||||||
|
RetentionCandidate(
|
||||||
|
ref=_stage_ref(stage.id),
|
||||||
|
kind="stage",
|
||||||
|
datasource_ref=(
|
||||||
|
_datasource_ref(stage.target_datasource_id)
|
||||||
|
if stage.target_datasource_id
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
disposition="delete_stage",
|
||||||
|
eligible_at=eligible_at,
|
||||||
|
eligible=not blockers,
|
||||||
|
blockers=blockers,
|
||||||
|
policy_version=str(policy["version"]),
|
||||||
|
policy_hash=canonical_hash(policy),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
materializations = session.execute(
|
||||||
|
select(DatasourceMaterializationRecord, DatasourceRecord)
|
||||||
|
.join(DatasourceRecord, DatasourceRecord.id == DatasourceMaterializationRecord.datasource_id)
|
||||||
|
.where(DatasourceMaterializationRecord.tenant_id == tenant_id)
|
||||||
|
)
|
||||||
|
for materialization, datasource in materializations:
|
||||||
|
if materialization.disposed_at is not None:
|
||||||
|
continue
|
||||||
|
snapshot = _mapping(materialization.governance_snapshot_)
|
||||||
|
policy = normalize_retention_policy(
|
||||||
|
_mapping(snapshot.get("retention_policy"))
|
||||||
|
)
|
||||||
|
duration_key = (
|
||||||
|
"frozen_evidence_days"
|
||||||
|
if materialization.frozen_at is not None
|
||||||
|
else "materialization_days"
|
||||||
|
)
|
||||||
|
duration = policy.get(duration_key)
|
||||||
|
if not policy["enabled"] or not isinstance(duration, int):
|
||||||
|
continue
|
||||||
|
eligible_at = _as_utc(materialization.created_at) + timedelta(days=duration)
|
||||||
|
if eligible_at > effective_at:
|
||||||
|
continue
|
||||||
|
blockers: list[str] = []
|
||||||
|
if datasource.current_materialization_id == materialization.id:
|
||||||
|
blockers.append("current_materialization")
|
||||||
|
if tuple(snapshot.get("hold_refs") or ()):
|
||||||
|
blockers.append("legal_hold")
|
||||||
|
publication_count = session.scalar(
|
||||||
|
select(func.count(DatasourcePublicationRecord.id)).where(
|
||||||
|
DatasourcePublicationRecord.materialization_id == materialization.id
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if publication_count:
|
||||||
|
blockers.append("publication_evidence")
|
||||||
|
candidates.append(
|
||||||
|
RetentionCandidate(
|
||||||
|
ref=_materialization_ref(materialization.id),
|
||||||
|
kind="materialization",
|
||||||
|
datasource_ref=_datasource_ref(datasource.id),
|
||||||
|
disposition="purge_materialization_payload",
|
||||||
|
eligible_at=eligible_at,
|
||||||
|
eligible=not blockers,
|
||||||
|
blockers=tuple(blockers),
|
||||||
|
policy_version=str(policy["version"]),
|
||||||
|
policy_hash=canonical_hash(policy),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
ordered = tuple(sorted(candidates, key=lambda item: (item.kind, item.ref)))
|
||||||
|
plan_payload = {
|
||||||
|
"tenant_id": tenant_id,
|
||||||
|
"as_of": effective_at.isoformat(),
|
||||||
|
"candidates": [item.to_dict() for item in ordered],
|
||||||
|
}
|
||||||
|
return RetentionPlan(
|
||||||
|
as_of=effective_at,
|
||||||
|
plan_hash=canonical_hash(plan_payload),
|
||||||
|
candidates=ordered,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def apply_retention_plan(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
actor_ref: str,
|
||||||
|
plan: RetentionPlan,
|
||||||
|
target_refs: Sequence[str],
|
||||||
|
) -> tuple[tuple[str, ...], tuple[str, ...]]:
|
||||||
|
by_ref = {item.ref: item for item in plan.candidates}
|
||||||
|
requested = tuple(dict.fromkeys(target_refs))
|
||||||
|
missing = [ref for ref in requested if ref not in by_ref]
|
||||||
|
if missing:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Retention targets are not in the current plan: {', '.join(missing)}."
|
||||||
|
)
|
||||||
|
blocked = [ref for ref in requested if not by_ref[ref].eligible]
|
||||||
|
if blocked:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Retention targets have active blockers: {', '.join(blocked)}."
|
||||||
|
)
|
||||||
|
disposed: list[str] = []
|
||||||
|
evidence_hashes: list[str] = []
|
||||||
|
for ref in requested:
|
||||||
|
candidate = by_ref[ref]
|
||||||
|
if candidate.kind == "stage":
|
||||||
|
stage = _stage_by_ref(session, tenant_id=tenant_id, ref=ref)
|
||||||
|
digest = stage_subject_digest(
|
||||||
|
stage,
|
||||||
|
policy=normalize_approval_policy(
|
||||||
|
_mapping(_mapping(stage.approval_).get("policy"))
|
||||||
|
),
|
||||||
|
)
|
||||||
|
evidence = record_lifecycle_evidence(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
subject_ref=ref,
|
||||||
|
event_type="retention.stage_deleted",
|
||||||
|
actor_ref=actor_ref,
|
||||||
|
subject_digest=digest,
|
||||||
|
policy_version=candidate.policy_version,
|
||||||
|
policy_hash=candidate.policy_hash,
|
||||||
|
details={
|
||||||
|
"plan_hash": plan.plan_hash,
|
||||||
|
"disposition": candidate.disposition,
|
||||||
|
"validation_policy_hash": stage.validation_.get("policy_hash"),
|
||||||
|
"approval_policy_hash": stage.approval_.get("policy_hash"),
|
||||||
|
},
|
||||||
|
occurred_at=plan.as_of,
|
||||||
|
)
|
||||||
|
evidence_hashes.append(evidence.event_hash)
|
||||||
|
session.delete(stage)
|
||||||
|
else:
|
||||||
|
materialization = _materialization_by_ref(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
ref=ref,
|
||||||
|
)
|
||||||
|
payload = materialization.payload
|
||||||
|
subject_digest = canonical_hash(
|
||||||
|
{
|
||||||
|
"ref": ref,
|
||||||
|
"fingerprint": materialization.fingerprint,
|
||||||
|
"payload_checksum": materialization.payload_checksum,
|
||||||
|
"revision": materialization.revision,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
evidence = record_lifecycle_evidence(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
subject_ref=ref,
|
||||||
|
event_type="retention.materialization_payload_purged",
|
||||||
|
actor_ref=actor_ref,
|
||||||
|
subject_digest=subject_digest,
|
||||||
|
policy_version=candidate.policy_version,
|
||||||
|
policy_hash=candidate.policy_hash,
|
||||||
|
details={
|
||||||
|
"plan_hash": plan.plan_hash,
|
||||||
|
"disposition": candidate.disposition,
|
||||||
|
"payload_checksum": materialization.payload_checksum,
|
||||||
|
"row_count": materialization.row_count,
|
||||||
|
"byte_count": materialization.byte_count,
|
||||||
|
"frozen": materialization.frozen_at is not None,
|
||||||
|
},
|
||||||
|
occurred_at=plan.as_of,
|
||||||
|
)
|
||||||
|
evidence_hashes.append(evidence.event_hash)
|
||||||
|
materialization.payload_id = None
|
||||||
|
materialization.rows = []
|
||||||
|
materialization.state = "disposed"
|
||||||
|
materialization.disposed_at = plan.as_of
|
||||||
|
materialization.disposition_ = {
|
||||||
|
"plan_hash": plan.plan_hash,
|
||||||
|
"policy_version": candidate.policy_version,
|
||||||
|
"policy_hash": candidate.policy_hash,
|
||||||
|
"actor_ref": actor_ref,
|
||||||
|
"evidence_hash": evidence.event_hash,
|
||||||
|
"payload_checksum": materialization.payload_checksum,
|
||||||
|
}
|
||||||
|
session.flush()
|
||||||
|
if payload is not None:
|
||||||
|
remaining = session.scalar(
|
||||||
|
select(func.count(DatasourceMaterializationRecord.id)).where(
|
||||||
|
DatasourceMaterializationRecord.payload_id == payload.id
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if not remaining:
|
||||||
|
session.delete(payload)
|
||||||
|
disposed.append(ref)
|
||||||
|
session.flush()
|
||||||
|
return tuple(disposed), tuple(evidence_hashes)
|
||||||
|
|
||||||
|
|
||||||
|
def _stage_by_ref(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
ref: str,
|
||||||
|
) -> DatasourceStageRecord:
|
||||||
|
item = session.scalar(
|
||||||
|
select(DatasourceStageRecord).where(
|
||||||
|
DatasourceStageRecord.id == _ref_id(ref, "stage"),
|
||||||
|
DatasourceStageRecord.tenant_id == tenant_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if item is None:
|
||||||
|
raise DatasourceValidationError("Datasource stage is no longer available.")
|
||||||
|
return item
|
||||||
|
|
||||||
|
|
||||||
|
def _materialization_by_ref(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
ref: str,
|
||||||
|
) -> DatasourceMaterializationRecord:
|
||||||
|
item = session.scalar(
|
||||||
|
select(DatasourceMaterializationRecord).where(
|
||||||
|
DatasourceMaterializationRecord.id == _ref_id(ref, "materialization"),
|
||||||
|
DatasourceMaterializationRecord.tenant_id == tenant_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if item is None:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"Datasource materialization is no longer available."
|
||||||
|
)
|
||||||
|
return item
|
||||||
|
|
||||||
|
|
||||||
|
def _mapping(value: object) -> Mapping[str, object]:
|
||||||
|
return value if isinstance(value, Mapping) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def _mapping_sequence(value: object) -> tuple[Mapping[str, object], ...]:
|
||||||
|
if not isinstance(value, Sequence) or isinstance(value, (str, bytes)):
|
||||||
|
return ()
|
||||||
|
return tuple(item for item in value if isinstance(item, Mapping))
|
||||||
|
|
||||||
|
|
||||||
|
def _boolean(value: object, *, label: str) -> bool:
|
||||||
|
if not isinstance(value, bool):
|
||||||
|
raise DatasourceValidationError(f"Approval or retention {label} must be boolean.")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _bounded_integer(
|
||||||
|
value: object,
|
||||||
|
*,
|
||||||
|
label: str,
|
||||||
|
minimum: int,
|
||||||
|
maximum: int,
|
||||||
|
) -> int:
|
||||||
|
if isinstance(value, bool) or not isinstance(value, int):
|
||||||
|
raise DatasourceValidationError(f"Policy {label} must be an integer.")
|
||||||
|
if value < minimum or value > maximum:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Policy {label} must be between {minimum} and {maximum}."
|
||||||
|
)
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _optional_duration(value: object, *, label: str) -> int | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
return _bounded_integer(value, label=label, minimum=1, maximum=36_500)
|
||||||
|
|
||||||
|
|
||||||
|
def _optional_text(value: object) -> str | None:
|
||||||
|
text = str(value or "").strip()
|
||||||
|
return text or None
|
||||||
|
|
||||||
|
|
||||||
|
def _optional_datetime(value: object) -> datetime | None:
|
||||||
|
if not value:
|
||||||
|
return None
|
||||||
|
if isinstance(value, datetime):
|
||||||
|
return _as_utc(value)
|
||||||
|
try:
|
||||||
|
return _as_utc(datetime.fromisoformat(str(value)))
|
||||||
|
except ValueError as exc:
|
||||||
|
raise DatasourceValidationError("Approval expiry is invalid.") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _as_utc(value: datetime) -> datetime:
|
||||||
|
if value.tzinfo is None:
|
||||||
|
return value.replace(tzinfo=UTC)
|
||||||
|
return value.astimezone(UTC)
|
||||||
|
|
||||||
|
|
||||||
|
def _ref_id(ref: str, prefix: str) -> str:
|
||||||
|
expected = f"{prefix}:"
|
||||||
|
if not ref.startswith(expected) or not ref.removeprefix(expected).strip():
|
||||||
|
raise DatasourceValidationError(f"Invalid {prefix} reference.")
|
||||||
|
return ref.removeprefix(expected)
|
||||||
|
|
||||||
|
|
||||||
|
def _stage_ref(value: str) -> str:
|
||||||
|
return f"stage:{value}"
|
||||||
|
|
||||||
|
|
||||||
|
def _datasource_ref(value: str) -> str:
|
||||||
|
return f"datasource:{value}"
|
||||||
|
|
||||||
|
|
||||||
|
def _materialization_ref(value: str) -> str:
|
||||||
|
return f"materialization:{value}"
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"RetentionCandidate",
|
||||||
|
"RetentionPlan",
|
||||||
|
"apply_retention_plan",
|
||||||
|
"build_retention_plan",
|
||||||
|
"canonical_hash",
|
||||||
|
"decide_stage",
|
||||||
|
"ensure_stage_approval_current",
|
||||||
|
"initialize_stage_approval",
|
||||||
|
"list_lifecycle_evidence",
|
||||||
|
"normalize_approval_policy",
|
||||||
|
"normalize_retention_policy",
|
||||||
|
"record_lifecycle_evidence",
|
||||||
|
"stage_subject_digest",
|
||||||
|
]
|
||||||
@@ -21,6 +21,7 @@ from govoplan_core.core.module_guards import (
|
|||||||
)
|
)
|
||||||
from govoplan_core.core.modules import (
|
from govoplan_core.core.modules import (
|
||||||
CapabilityDocumentation,
|
CapabilityDocumentation,
|
||||||
|
DocumentationCondition,
|
||||||
DocumentationLink,
|
DocumentationLink,
|
||||||
DocumentationTopic,
|
DocumentationTopic,
|
||||||
FrontendModule,
|
FrontendModule,
|
||||||
@@ -55,6 +56,7 @@ from govoplan_datasources.backend.service import (
|
|||||||
ADMIN_SCOPE,
|
ADMIN_SCOPE,
|
||||||
CATALOGUE_READ_SCOPE,
|
CATALOGUE_READ_SCOPE,
|
||||||
SOURCE_WRITE_SCOPE,
|
SOURCE_WRITE_SCOPE,
|
||||||
|
STAGE_APPROVE_SCOPE,
|
||||||
STAGE_WRITE_SCOPE,
|
STAGE_WRITE_SCOPE,
|
||||||
SqlDatasourceProvider,
|
SqlDatasourceProvider,
|
||||||
)
|
)
|
||||||
@@ -63,7 +65,7 @@ from govoplan_datasources.backend.payloads import ExternalArtifactPayloadBackend
|
|||||||
|
|
||||||
MODULE_ID = "datasources"
|
MODULE_ID = "datasources"
|
||||||
MODULE_NAME = "Datasources"
|
MODULE_NAME = "Datasources"
|
||||||
MODULE_VERSION = "0.1.20"
|
MODULE_VERSION = "0.1.21"
|
||||||
DATASOURCE_INTERFACE_VERSION = "0.2.0"
|
DATASOURCE_INTERFACE_VERSION = "0.2.0"
|
||||||
|
|
||||||
ARCHITECTURE = ModuleArchitectureDeclaration(
|
ARCHITECTURE = ModuleArchitectureDeclaration(
|
||||||
@@ -149,6 +151,11 @@ PERMISSIONS = (
|
|||||||
"Stage datasource content",
|
"Stage datasource content",
|
||||||
"Upload, validate, inspect, and promote bounded datasource stages.",
|
"Upload, validate, inspect, and promote bounded datasource stages.",
|
||||||
),
|
),
|
||||||
|
_permission(
|
||||||
|
STAGE_APPROVE_SCOPE,
|
||||||
|
"Approve datasource promotion",
|
||||||
|
"Review staged validation evidence and record an attributable promotion decision.",
|
||||||
|
),
|
||||||
_permission(
|
_permission(
|
||||||
ADMIN_SCOPE,
|
ADMIN_SCOPE,
|
||||||
"Administer datasources",
|
"Administer datasources",
|
||||||
@@ -167,6 +174,12 @@ ROLE_TEMPLATES = (
|
|||||||
STAGE_WRITE_SCOPE,
|
STAGE_WRITE_SCOPE,
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
RoleTemplate(
|
||||||
|
slug="datasource_approver",
|
||||||
|
name="Datasource approver",
|
||||||
|
description="Independently approve or reject governed datasource stages.",
|
||||||
|
permissions=(CATALOGUE_READ_SCOPE, STAGE_APPROVE_SCOPE),
|
||||||
|
),
|
||||||
RoleTemplate(
|
RoleTemplate(
|
||||||
slug="datasource_reader",
|
slug="datasource_reader",
|
||||||
name="Datasource reader",
|
name="Datasource reader",
|
||||||
@@ -235,6 +248,15 @@ def _tenant_summary(session, tenant_id: str) -> dict[str, int]:
|
|||||||
)
|
)
|
||||||
.count()
|
.count()
|
||||||
),
|
),
|
||||||
|
"datasource_stages_awaiting_approval": (
|
||||||
|
session.query(datasource_models.DatasourceStageRecord)
|
||||||
|
.filter(
|
||||||
|
datasource_models.DatasourceStageRecord.tenant_id == tenant_id,
|
||||||
|
datasource_models.DatasourceStageRecord.state
|
||||||
|
== "awaiting_approval",
|
||||||
|
)
|
||||||
|
.count()
|
||||||
|
),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +412,20 @@ manifest = ModuleManifest(
|
|||||||
label="Datasource preview and materializations",
|
label="Datasource preview and materializations",
|
||||||
order=50,
|
order=50,
|
||||||
),
|
),
|
||||||
|
ViewSurface(
|
||||||
|
id="datasources.lifecycle-evidence",
|
||||||
|
module_id=MODULE_ID,
|
||||||
|
kind="section",
|
||||||
|
label="Datasource lifecycle evidence",
|
||||||
|
order=60,
|
||||||
|
),
|
||||||
|
ViewSurface(
|
||||||
|
id="datasources.retention",
|
||||||
|
module_id=MODULE_ID,
|
||||||
|
kind="action",
|
||||||
|
label="Datasource retention preview and apply",
|
||||||
|
order=70,
|
||||||
|
),
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
route_factory=_router,
|
route_factory=_router,
|
||||||
@@ -421,6 +457,7 @@ manifest = ModuleManifest(
|
|||||||
script_location=str(Path(__file__).with_name("migrations") / "versions"),
|
script_location=str(Path(__file__).with_name("migrations") / "versions"),
|
||||||
retirement_supported=True,
|
retirement_supported=True,
|
||||||
retirement_provider=drop_table_retirement_provider(
|
retirement_provider=drop_table_retirement_provider(
|
||||||
|
datasource_models.DatasourceLifecycleEvidenceRecord,
|
||||||
datasource_models.DatasourcePublicationRecord,
|
datasource_models.DatasourcePublicationRecord,
|
||||||
datasource_models.DatasourceStageRecord,
|
datasource_models.DatasourceStageRecord,
|
||||||
datasource_models.DatasourceMaterializationRecord,
|
datasource_models.DatasourceMaterializationRecord,
|
||||||
@@ -436,6 +473,7 @@ manifest = ModuleManifest(
|
|||||||
),
|
),
|
||||||
uninstall_guard_providers=(
|
uninstall_guard_providers=(
|
||||||
persistent_table_uninstall_guard(
|
persistent_table_uninstall_guard(
|
||||||
|
datasource_models.DatasourceLifecycleEvidenceRecord,
|
||||||
datasource_models.DatasourceRecord,
|
datasource_models.DatasourceRecord,
|
||||||
datasource_models.DatasourceMaterializationRecord,
|
datasource_models.DatasourceMaterializationRecord,
|
||||||
datasource_models.DatasourcePayloadRecord,
|
datasource_models.DatasourcePayloadRecord,
|
||||||
@@ -452,7 +490,7 @@ manifest = ModuleManifest(
|
|||||||
title="Datasources data-subject requests",
|
title="Datasources data-subject requests",
|
||||||
summary="Identify governed datasource copies while preserving credential, payload, and immutable-evidence boundaries.",
|
summary="Identify governed datasource copies while preserving credential, payload, and immutable-evidence boundaries.",
|
||||||
body=(
|
body=(
|
||||||
"Datasources matches exact tenant-scoped catalogue, governance-reference, materialization, payload, stage, and publication identifiers plus minimized account, identity, or membership operator attribution. DSAR results never copy connector/provider references, locators, credentials, arbitrary rows, schemas, validation samples, metadata, provenance bodies, checkpoints, idempotency material, or hashes. Arbitrary tabular payloads are not scanned for identifiers because that would be incomplete, schema-dependent, and liable to disclose unrelated people; the authoritative source module must locate and correct subject facts. "
|
"Datasources matches exact tenant-scoped catalogue, governance-reference, materialization, payload, stage, publication, and lifecycle-evidence identifiers plus minimized account, identity, or membership operator attribution. DSAR results never copy connector/provider references, locators, credentials, arbitrary rows, schemas, validation samples, metadata, provenance bodies, checkpoints, idempotency material, or hashes. Arbitrary tabular payloads are not scanned for identifiers because that would be incomplete, schema-dependent, and liable to disclose unrelated people; the authoritative source module must locate and correct subject facts. "
|
||||||
"An unpromoted stage or unreferenced payload can be deleted idempotently. Published catalogue state, promoted stages, referenced payloads, immutable materializations, governance references, publications, holds, and operator attribution require data-steward review and source correction. Downstream Dataflow and Reporting outputs must be refreshed after correction."
|
"An unpromoted stage or unreferenced payload can be deleted idempotently. Published catalogue state, promoted stages, referenced payloads, immutable materializations, governance references, publications, holds, and operator attribution require data-steward review and source correction. Downstream Dataflow and Reporting outputs must be refreshed after correction."
|
||||||
),
|
),
|
||||||
layer="configured",
|
layer="configured",
|
||||||
@@ -524,8 +562,8 @@ manifest = ModuleManifest(
|
|||||||
"Authority mode states whether GovOPlaN, an external system, a synchronized projection, an overlay, or a linked reference "
|
"Authority mode states whether GovOPlaN, an external system, a synchronized projection, an overlay, or a linked reference "
|
||||||
"controls the data. The authoritative source, owner, steward, responsible organization/function, schema owner, privacy "
|
"controls the data. The authoritative source, owner, steward, responsible organization/function, schema owner, privacy "
|
||||||
"profile, retention policy, transfer agreement, legal basis, holds, correction procedure, purposes, official keys, and "
|
"profile, retention policy, transfer agreement, legal basis, holds, correction procedure, purposes, official keys, and "
|
||||||
"known limits provide discoverable institutional context. A retention-policy reference identifies the owning Policy rule; it "
|
"known limits provide discoverable institutional context. A retention-policy reference identifies an owning external Policy rule, while "
|
||||||
"does not itself delete materializations, override legal holds, or prove that a scheduler is active. A transfer-agreement "
|
"the local versioned retention contract controls previewable stage and payload disposition. Retention never overrides legal holds, the current materialization, or publication evidence. A transfer-agreement "
|
||||||
"reference records the governed agreement for external exchange; it does not grant connector credentials, recipient access, "
|
"reference records the governed agreement for external exchange; it does not grant connector credentials, recipient access, "
|
||||||
"or authority to export classified rows. Freshness and quality policies are typed JSON contracts retained "
|
"or authority to export classified rows. Freshness and quality policies are typed JSON contracts retained "
|
||||||
"with materialization evidence. Datasources enforces the declared bounded tabular stage rules and schema policy; origin-specific "
|
"with materialization evidence. Datasources enforces the declared bounded tabular stage rules and schema policy; origin-specific "
|
||||||
@@ -603,8 +641,7 @@ manifest = ModuleManifest(
|
|||||||
"durable provider-neutral artifact reference with a pinned locator, SHA-256 checksum, declared schema, fingerprint, and size. "
|
"durable provider-neutral artifact reference with a pinned locator, SHA-256 checksum, declared schema, fingerprint, and size. "
|
||||||
"The configured payload backend verifies the artifact and provides bounded reads. Content-level rules require checksum- and "
|
"The configured payload backend verifies the artifact and provides bounded reads. Content-level rules require checksum- and "
|
||||||
"policy-bound producer evidence; absent evidence creates an immutable review-required materialization without changing the "
|
"policy-bound producer evidence; absent evidence creates an immutable review-required materialization without changing the "
|
||||||
"Datasource's current state. Warning and review-required outcomes are preserved for Workflow handoffs. Approval and retention execution "
|
"Datasource's current state. Warning and review-required outcomes are preserved for Workflow handoffs. Versioned approval policy can require an attributable, separated quorum before a stage or refresh becomes current; retention uses a fresh, hashed preview before any explicit administrator apply."
|
||||||
"are not inferred from arbitrary JSON flags and remain separate governed lifecycle work."
|
|
||||||
),
|
),
|
||||||
layer="available",
|
layer="available",
|
||||||
documentation_types=("admin", "user"),
|
documentation_types=("admin", "user"),
|
||||||
@@ -634,7 +671,8 @@ manifest = ModuleManifest(
|
|||||||
],
|
],
|
||||||
"limitations": [
|
"limitations": [
|
||||||
"Referential rules currently use a bounded embedded value set rather than reading another protected Datasource.",
|
"Referential rules currently use a bounded embedded value set rather than reading another protected Datasource.",
|
||||||
"Approval authority and automatic retention execution are not part of the current stage contract.",
|
"Approvals are module-local lifecycle evidence; installing a separate Approvals module does not silently change configured datasource policy.",
|
||||||
|
"Retention is never automatic: an administrator must preview a current plan and explicitly apply eligible targets.",
|
||||||
"Artifact bytes remain owned by their payload backend; Datasources stores an immutable reference and integrity evidence.",
|
"Artifact bytes remain owned by their payload backend; Datasources stores an immutable reference and integrity evidence.",
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
@@ -647,7 +685,7 @@ manifest = ModuleManifest(
|
|||||||
"A Datasource key is the stable catalogue identity used by consumers. Live mode reads through an available origin; cached "
|
"A Datasource key is the stable catalogue identity used by consumers. Live mode reads through an available origin; cached "
|
||||||
"mode refreshes an origin into immutable revisions; static mode promotes uploaded content from staging. Stages are bounded, "
|
"mode refreshes an origin into immutable revisions; static mode promotes uploaded content from staging. Stages are bounded, "
|
||||||
"inspectable, and non-consumable until promoted. Promotion creates or updates a governed Datasource and appends an immutable "
|
"inspectable, and non-consumable until promoted. Promotion creates or updates a governed Datasource and appends an immutable "
|
||||||
"materialization. Refresh appends a new cached revision without rewriting older evidence. Freeze labels an immutable, "
|
"materialization. When approval is configured, a refresh first creates a non-consumable stage and only an approved quorum permits promotion. Freeze labels an immutable, "
|
||||||
"addressable state for reproducible execution. Retirement removes the Datasource from new definitions while retained "
|
"addressable state for reproducible execution. Retirement removes the Datasource from new definitions while retained "
|
||||||
"materialization references remain governed. Connector absence disables origin registration but leaves local catalogue and "
|
"materialization references remain governed. Connector absence disables origin registration but leaves local catalogue and "
|
||||||
"staging behavior available."
|
"staging behavior available."
|
||||||
@@ -682,6 +720,56 @@ manifest = ModuleManifest(
|
|||||||
},
|
},
|
||||||
},
|
},
|
||||||
),
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="datasources.approval-and-retention",
|
||||||
|
title="Approve promotions and apply retention safely",
|
||||||
|
summary="Require separated datasource decisions and preview retention consequences before disposing staged or materialized payloads.",
|
||||||
|
body=(
|
||||||
|
"A versioned approval policy can require one to five distinct approvals before a valid stage becomes ready. The stage creator cannot approve when separation of duties is enabled. Decisions are bound to the exact stage fingerprint, quality-policy hash, approval-policy hash, actor authority, reason, and expiry; a changed policy or staged subject invalidates promotion. Cached refreshes use the same stage path whenever approval is required. "
|
||||||
|
"A versioned retention policy independently sets durations for stages, ordinary materializations, and frozen evidence. Preview returns a hashed plan with every due target and blocker. Current materializations, legal holds, pending approvals, and producer-publication evidence cannot be disposed. Applying the unchanged plan deletes eligible transient stages or purges materialization payload rows while retaining minimized schema/provenance, checksum, disposition, and hash-chained lifecycle evidence. Empty local policies disable both gates, so the module remains usable without Access or Policy; installed modules may tighten access but do not manufacture approval claims."
|
||||||
|
),
|
||||||
|
layer="available",
|
||||||
|
documentation_types=("admin", "user"),
|
||||||
|
audience=("operator", "module_admin", "data_steward", "auditor"),
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
any_scopes=(
|
||||||
|
STAGE_WRITE_SCOPE,
|
||||||
|
STAGE_APPROVE_SCOPE,
|
||||||
|
ADMIN_SCOPE,
|
||||||
|
)
|
||||||
|
),
|
||||||
|
),
|
||||||
|
related_modules=("access", "audit", "policy", "workflow_engine"),
|
||||||
|
order=75,
|
||||||
|
translations={
|
||||||
|
"de": {
|
||||||
|
"title": "Freigaben erteilen und Aufbewahrung sicher anwenden",
|
||||||
|
"summary": "Getrennte Entscheidungen für Datenquellen verlangen und Folgen der Aufbewahrung vor dem Löschen von Stufen oder Nutzdaten prüfen.",
|
||||||
|
"body": (
|
||||||
|
"Eine versionierte Freigaberichtlinie kann ein bis fünf verschiedene Freigaben verlangen, bevor eine gültige Stufe bereit ist. Bei aktivierter Funktionstrennung darf die erstellende Person nicht selbst freigeben. Entscheidungen sind an Fingerabdruck, Qualitäts- und Freigaberichtlinien-Hash, Berechtigung, Begründung und Ablaufzeit gebunden; geänderte Richtlinien oder Inhalte verhindern die Übernahme. Zwischengespeicherte Aktualisierungen nutzen bei Freigabepflicht denselben Stufenweg. "
|
||||||
|
"Eine getrennte Aufbewahrungsrichtlinie legt Fristen für Stufen, gewöhnliche Materialisierungen und eingefrorene Nachweise fest. Die Vorschau liefert einen gehashten Plan mit fälligen Zielen und Sperrgründen. Aktuelle Materialisierungen, rechtliche Sperren, offene Freigaben und Veröffentlichungsnachweise dürfen nicht entfernt werden. Beim Anwenden des unveränderten Plans werden nur zulässige Stufen oder Nutzdaten gelöscht; minimierte Metadaten, Prüfsummen, Dispositionsangaben und hashverkettete Lebenszyklusnachweise bleiben erhalten. Leere lokale Richtlinien deaktivieren beide Schranken, sodass das Modul ohne Access oder Policy nutzbar bleibt."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
},
|
||||||
|
metadata={
|
||||||
|
"seed": True,
|
||||||
|
"kind": "workflow",
|
||||||
|
"help_contexts": [
|
||||||
|
"datasources.staging.approval",
|
||||||
|
"datasources.action.approve",
|
||||||
|
"datasources.action.reject",
|
||||||
|
"datasources.field.approval-policy",
|
||||||
|
"datasources.field.retention-policy-contract",
|
||||||
|
"datasources.lifecycle-evidence",
|
||||||
|
"datasources.retention",
|
||||||
|
],
|
||||||
|
"limitations": [
|
||||||
|
"Retention execution is explicit and plan-bound; this module does not run a hidden deletion scheduler.",
|
||||||
|
"Disposed materialization records retain minimized schema and provenance metadata while their payload rows are removed.",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
),
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
+142
@@ -0,0 +1,142 @@
|
|||||||
|
"""v0.1.21 datasource lifecycle governance
|
||||||
|
|
||||||
|
Revision ID: d1a7c3e9f5b2
|
||||||
|
Revises: c9e3a6f1d4b8
|
||||||
|
Create Date: 2026-08-22 20:00:00.000000
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision = "d1a7c3e9f5b2"
|
||||||
|
down_revision = "c9e3a6f1d4b8"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.add_column(
|
||||||
|
"datasource_catalogue",
|
||||||
|
sa.Column(
|
||||||
|
"approval_policy",
|
||||||
|
sa.JSON(),
|
||||||
|
server_default=sa.text("'{}'"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.add_column(
|
||||||
|
"datasource_catalogue",
|
||||||
|
sa.Column(
|
||||||
|
"retention_policy",
|
||||||
|
sa.JSON(),
|
||||||
|
server_default=sa.text("'{}'"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.add_column(
|
||||||
|
"datasource_stages",
|
||||||
|
sa.Column(
|
||||||
|
"approval",
|
||||||
|
sa.JSON(),
|
||||||
|
server_default=sa.text("'{}'"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.add_column(
|
||||||
|
"datasource_materializations",
|
||||||
|
sa.Column("disposed_at", sa.DateTime(timezone=True), nullable=True),
|
||||||
|
)
|
||||||
|
op.add_column(
|
||||||
|
"datasource_materializations",
|
||||||
|
sa.Column(
|
||||||
|
"disposition",
|
||||||
|
sa.JSON(),
|
||||||
|
server_default=sa.text("'{}'"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_materializations_disposed_at",
|
||||||
|
"datasource_materializations",
|
||||||
|
["disposed_at"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_table(
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
sa.Column("id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("subject_ref", sa.String(length=160), nullable=False),
|
||||||
|
sa.Column("event_type", sa.String(length=80), nullable=False),
|
||||||
|
sa.Column("occurred_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("actor_ref", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("policy_version", sa.String(length=120), nullable=True),
|
||||||
|
sa.Column("policy_hash", sa.String(length=64), nullable=True),
|
||||||
|
sa.Column("subject_digest", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column("previous_event_hash", sa.String(length=64), nullable=True),
|
||||||
|
sa.Column("event_hash", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column("details", sa.JSON(), nullable=False),
|
||||||
|
sa.PrimaryKeyConstraint("id"),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"event_hash",
|
||||||
|
name="uq_datasource_lifecycle_evidence_hash",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_lifecycle_evidence_tenant_id",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
["tenant_id"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_lifecycle_evidence_subject_ref",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
["subject_ref"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_lifecycle_evidence_event_type",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
["event_type"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_lifecycle_evidence_actor_ref",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
["actor_ref"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_lifecycle_evidence_event_hash",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
["event_hash"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_lifecycle_evidence_subject",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
["tenant_id", "subject_ref", "occurred_at"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_lifecycle_evidence_event",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
["tenant_id", "event_type", "occurred_at"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_table("datasource_lifecycle_evidence")
|
||||||
|
op.drop_index(
|
||||||
|
"ix_datasource_materializations_disposed_at",
|
||||||
|
table_name="datasource_materializations",
|
||||||
|
)
|
||||||
|
op.drop_column("datasource_materializations", "disposition")
|
||||||
|
op.drop_column("datasource_materializations", "disposed_at")
|
||||||
|
op.drop_column("datasource_stages", "approval")
|
||||||
|
op.drop_column("datasource_catalogue", "retention_policy")
|
||||||
|
op.drop_column("datasource_catalogue", "approval_policy")
|
||||||
@@ -3,6 +3,7 @@ from __future__ import annotations
|
|||||||
import hashlib
|
import hashlib
|
||||||
import json
|
import json
|
||||||
from collections.abc import Mapping
|
from collections.abc import Mapping
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
@@ -30,6 +31,8 @@ from govoplan_datasources.backend.schemas import (
|
|||||||
DatasourceFreezeRequest,
|
DatasourceFreezeRequest,
|
||||||
DatasourceGovernancePayload,
|
DatasourceGovernancePayload,
|
||||||
DatasourceGovernanceUpdateRequest,
|
DatasourceGovernanceUpdateRequest,
|
||||||
|
DatasourceLifecycleEvidenceListResponse,
|
||||||
|
DatasourceLifecycleEvidenceResponse,
|
||||||
DatasourceListResponse,
|
DatasourceListResponse,
|
||||||
DatasourceMaterializationListResponse,
|
DatasourceMaterializationListResponse,
|
||||||
DatasourceMaterializationResponse,
|
DatasourceMaterializationResponse,
|
||||||
@@ -41,8 +44,13 @@ from govoplan_datasources.backend.schemas import (
|
|||||||
DatasourcePreviewDiagnosticResponse,
|
DatasourcePreviewDiagnosticResponse,
|
||||||
DatasourcePreviewResponse,
|
DatasourcePreviewResponse,
|
||||||
DatasourceResponse,
|
DatasourceResponse,
|
||||||
|
DatasourceRetentionApplyRequest,
|
||||||
|
DatasourceRetentionApplyResponse,
|
||||||
|
DatasourceRetentionCandidateResponse,
|
||||||
|
DatasourceRetentionPlanResponse,
|
||||||
DatasourceRetireResponse,
|
DatasourceRetireResponse,
|
||||||
DatasourceStageCreateRequest,
|
DatasourceStageCreateRequest,
|
||||||
|
DatasourceStageDecisionRequest,
|
||||||
DatasourceStageListResponse,
|
DatasourceStageListResponse,
|
||||||
DatasourceStagePromoteRequest,
|
DatasourceStagePromoteRequest,
|
||||||
DatasourceStagePromoteResponse,
|
DatasourceStagePromoteResponse,
|
||||||
@@ -52,6 +60,7 @@ from govoplan_datasources.backend.service import (
|
|||||||
ADMIN_SCOPE,
|
ADMIN_SCOPE,
|
||||||
CATALOGUE_READ_SCOPE,
|
CATALOGUE_READ_SCOPE,
|
||||||
SOURCE_WRITE_SCOPE,
|
SOURCE_WRITE_SCOPE,
|
||||||
|
STAGE_APPROVE_SCOPE,
|
||||||
STAGE_WRITE_SCOPE,
|
STAGE_WRITE_SCOPE,
|
||||||
SqlDatasourceProvider,
|
SqlDatasourceProvider,
|
||||||
)
|
)
|
||||||
@@ -236,6 +245,56 @@ def api_create_stage(
|
|||||||
"schema_classification": _safe_mapping(
|
"schema_classification": _safe_mapping(
|
||||||
stage.validation.get("schema_change")
|
stage.validation.get("schema_change")
|
||||||
).get("classification"),
|
).get("classification"),
|
||||||
|
"approval_state": stage.approval.get("state"),
|
||||||
|
"approval_policy_hash": stage.approval.get("policy_hash"),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return _stage_response(stage)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/stages/{stage_id}/decision",
|
||||||
|
response_model=DatasourceStageResponse,
|
||||||
|
)
|
||||||
|
def api_decide_stage(
|
||||||
|
stage_id: str,
|
||||||
|
payload: DatasourceStageDecisionRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> DatasourceStageResponse:
|
||||||
|
_require_any_scope(principal, STAGE_APPROVE_SCOPE, ADMIN_SCOPE)
|
||||||
|
try:
|
||||||
|
stage, evidence_hash, replayed = _provider().decide_stage(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
stage_ref=f"stage:{stage_id}",
|
||||||
|
decision=payload.decision,
|
||||||
|
reason=payload.reason,
|
||||||
|
expected_policy_hash=payload.expected_policy_hash,
|
||||||
|
expected_subject_digest=payload.expected_subject_digest,
|
||||||
|
)
|
||||||
|
except DatasourceError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
action = (
|
||||||
|
"datasources.stage.approved"
|
||||||
|
if payload.decision == "approve"
|
||||||
|
else "datasources.stage.rejected"
|
||||||
|
)
|
||||||
|
_audit(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action=action,
|
||||||
|
object_type="datasource_stage",
|
||||||
|
object_id=stage.ref,
|
||||||
|
details={
|
||||||
|
"decision": payload.decision,
|
||||||
|
"resulting_state": stage.state,
|
||||||
|
"approval_count": stage.approval.get("approval_count"),
|
||||||
|
"required_approvals": stage.approval.get("required_approvals"),
|
||||||
|
"policy_hash": stage.approval.get("policy_hash"),
|
||||||
|
"evidence_hash": evidence_hash,
|
||||||
|
"replayed": replayed,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
session.commit()
|
session.commit()
|
||||||
@@ -277,6 +336,12 @@ def api_promote_stage(
|
|||||||
"quality_policy_hash": _safe_mapping(
|
"quality_policy_hash": _safe_mapping(
|
||||||
materialization.provenance.get("stage_validation")
|
materialization.provenance.get("stage_validation")
|
||||||
).get("policy_hash"),
|
).get("policy_hash"),
|
||||||
|
"approval_policy_hash": _safe_mapping(
|
||||||
|
materialization.provenance.get("stage_approval")
|
||||||
|
).get("policy_hash"),
|
||||||
|
"promotion_evidence_hash": materialization.provenance.get(
|
||||||
|
"promotion_evidence_hash"
|
||||||
|
),
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
session.commit()
|
session.commit()
|
||||||
@@ -286,6 +351,100 @@ def api_promote_stage(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/lifecycle-evidence",
|
||||||
|
response_model=DatasourceLifecycleEvidenceListResponse,
|
||||||
|
)
|
||||||
|
def api_list_lifecycle_evidence(
|
||||||
|
subject_ref: str | None = Query(default=None, max_length=160),
|
||||||
|
limit: int = Query(default=200, ge=1, le=500),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> DatasourceLifecycleEvidenceListResponse:
|
||||||
|
_require_any_scope(principal, CATALOGUE_READ_SCOPE, ADMIN_SCOPE)
|
||||||
|
try:
|
||||||
|
rows = _provider().list_lifecycle_evidence(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
subject_ref=subject_ref,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
except DatasourceError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
return DatasourceLifecycleEvidenceListResponse(
|
||||||
|
evidence=[_evidence_response(item) for item in rows]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/retention/plan",
|
||||||
|
response_model=DatasourceRetentionPlanResponse,
|
||||||
|
)
|
||||||
|
def api_preview_retention(
|
||||||
|
as_of: datetime | None = Query(default=None),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> DatasourceRetentionPlanResponse:
|
||||||
|
_require_any_scope(principal, ADMIN_SCOPE)
|
||||||
|
effective_at = as_of or datetime.now(UTC)
|
||||||
|
try:
|
||||||
|
plan = _provider().preview_retention(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
as_of=effective_at,
|
||||||
|
)
|
||||||
|
except DatasourceError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
return DatasourceRetentionPlanResponse(
|
||||||
|
as_of=plan.as_of.isoformat(),
|
||||||
|
plan_hash=plan.plan_hash,
|
||||||
|
candidates=[
|
||||||
|
DatasourceRetentionCandidateResponse(**item.to_dict())
|
||||||
|
for item in plan.candidates
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/retention/apply",
|
||||||
|
response_model=DatasourceRetentionApplyResponse,
|
||||||
|
)
|
||||||
|
def api_apply_retention(
|
||||||
|
payload: DatasourceRetentionApplyRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> DatasourceRetentionApplyResponse:
|
||||||
|
_require_any_scope(principal, ADMIN_SCOPE)
|
||||||
|
try:
|
||||||
|
disposed, evidence_hashes = _provider().apply_retention(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
as_of=payload.as_of,
|
||||||
|
plan_hash=payload.plan_hash,
|
||||||
|
target_refs=payload.target_refs,
|
||||||
|
)
|
||||||
|
except DatasourceError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
_audit(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="datasources.retention.applied",
|
||||||
|
object_type="datasource_retention_plan",
|
||||||
|
object_id=payload.plan_hash,
|
||||||
|
details={
|
||||||
|
"as_of": payload.as_of.isoformat(),
|
||||||
|
"disposed_refs": list(disposed),
|
||||||
|
"evidence_hashes": list(evidence_hashes),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return DatasourceRetentionApplyResponse(
|
||||||
|
plan_hash=payload.plan_hash,
|
||||||
|
disposed_refs=list(disposed),
|
||||||
|
evidence_hashes=list(evidence_hashes),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.get("", response_model=DatasourceListResponse)
|
@router.get("", response_model=DatasourceListResponse)
|
||||||
def api_list_datasources(
|
def api_list_datasources(
|
||||||
query: str = Query(default="", max_length=200),
|
query: str = Query(default="", max_length=200),
|
||||||
@@ -507,6 +666,46 @@ def api_refresh_datasource(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/{datasource_id}/refresh/stage",
|
||||||
|
response_model=DatasourceStageResponse,
|
||||||
|
status_code=status.HTTP_201_CREATED,
|
||||||
|
)
|
||||||
|
def api_prepare_refresh(
|
||||||
|
datasource_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> DatasourceStageResponse:
|
||||||
|
_require_any_scope(
|
||||||
|
principal,
|
||||||
|
SOURCE_WRITE_SCOPE,
|
||||||
|
STAGE_WRITE_SCOPE,
|
||||||
|
ADMIN_SCOPE,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
stage = _provider().prepare_refresh(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
datasource_ref=f"datasource:{datasource_id}",
|
||||||
|
)
|
||||||
|
except DatasourceError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
_audit(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="datasources.refresh.staged",
|
||||||
|
object_type="datasource_stage",
|
||||||
|
object_id=stage.ref,
|
||||||
|
details={
|
||||||
|
"datasource_ref": stage.target_datasource_ref,
|
||||||
|
"approval_state": stage.approval.get("state"),
|
||||||
|
"quality_policy_hash": stage.validation.get("policy_hash"),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return _stage_response(stage)
|
||||||
|
|
||||||
|
|
||||||
@router.post(
|
@router.post(
|
||||||
"/{datasource_id}/freeze",
|
"/{datasource_id}/freeze",
|
||||||
response_model=DatasourceMaterializationResponse,
|
response_model=DatasourceMaterializationResponse,
|
||||||
@@ -635,6 +834,8 @@ def _materialization_response(
|
|||||||
item.source_timestamp.isoformat() if item.source_timestamp else None
|
item.source_timestamp.isoformat() if item.source_timestamp else None
|
||||||
),
|
),
|
||||||
created_at=item.created_at.isoformat() if item.created_at else None,
|
created_at=item.created_at.isoformat() if item.created_at else None,
|
||||||
|
disposed_at=item.disposed_at.isoformat() if item.disposed_at else None,
|
||||||
|
disposition=dict(item.disposition),
|
||||||
provenance=dict(item.provenance),
|
provenance=dict(item.provenance),
|
||||||
metadata=dict(item.metadata),
|
metadata=dict(item.metadata),
|
||||||
governance=item.governance.to_dict(),
|
governance=item.governance.to_dict(),
|
||||||
@@ -664,6 +865,7 @@ def _stage_response(item: DatasourceStage) -> DatasourceStageResponse:
|
|||||||
row_count=item.row_count,
|
row_count=item.row_count,
|
||||||
byte_count=item.byte_count,
|
byte_count=item.byte_count,
|
||||||
validation=dict(item.validation),
|
validation=dict(item.validation),
|
||||||
|
approval=dict(item.approval),
|
||||||
created_at=item.created_at.isoformat() if item.created_at else None,
|
created_at=item.created_at.isoformat() if item.created_at else None,
|
||||||
promoted_at=item.promoted_at.isoformat() if item.promoted_at else None,
|
promoted_at=item.promoted_at.isoformat() if item.promoted_at else None,
|
||||||
promoted_materialization_ref=item.promoted_materialization_ref,
|
promoted_materialization_ref=item.promoted_materialization_ref,
|
||||||
@@ -719,6 +921,22 @@ def _origin_response(item: DatasourceOrigin) -> DatasourceOriginResponse:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _evidence_response(item) -> DatasourceLifecycleEvidenceResponse:
|
||||||
|
return DatasourceLifecycleEvidenceResponse(
|
||||||
|
ref=f"lifecycle-evidence:{item.id}",
|
||||||
|
subject_ref=item.subject_ref,
|
||||||
|
event_type=item.event_type,
|
||||||
|
occurred_at=item.occurred_at.isoformat(),
|
||||||
|
actor_ref=item.actor_ref,
|
||||||
|
policy_version=item.policy_version,
|
||||||
|
policy_hash=item.policy_hash,
|
||||||
|
subject_digest=item.subject_digest,
|
||||||
|
previous_event_hash=item.previous_event_hash,
|
||||||
|
event_hash=item.event_hash,
|
||||||
|
details=dict(item.details_),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _governance(
|
def _governance(
|
||||||
payload: DatasourceGovernancePayload | None,
|
payload: DatasourceGovernancePayload | None,
|
||||||
) -> DatasourceGovernance | None:
|
) -> DatasourceGovernance | None:
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
from typing import Any, Literal
|
from typing import Any, Literal
|
||||||
|
|
||||||
from pydantic import BaseModel, Field, model_validator
|
from pydantic import BaseModel, Field, model_validator
|
||||||
@@ -50,6 +51,8 @@ class DatasourceGovernancePayload(BaseModel):
|
|||||||
transfer_agreement_ref: str | None = Field(default=None, max_length=500)
|
transfer_agreement_ref: str | None = Field(default=None, max_length=500)
|
||||||
freshness_policy: dict[str, Any] = Field(default_factory=dict)
|
freshness_policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
quality_policy: dict[str, Any] = Field(default_factory=dict)
|
quality_policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
approval_policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
retention_policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
known_limits: list[str] = Field(default_factory=list, max_length=100)
|
known_limits: list[str] = Field(default_factory=list, max_length=100)
|
||||||
correction_procedure_ref: str | None = Field(default=None, max_length=500)
|
correction_procedure_ref: str | None = Field(default=None, max_length=500)
|
||||||
affected_refs: list[str] = Field(default_factory=list, max_length=250)
|
affected_refs: list[str] = Field(default_factory=list, max_length=250)
|
||||||
@@ -116,6 +119,8 @@ class DatasourceMaterializationResponse(BaseModel):
|
|||||||
frozen_label: str | None
|
frozen_label: str | None
|
||||||
source_timestamp: str | None
|
source_timestamp: str | None
|
||||||
created_at: str | None
|
created_at: str | None
|
||||||
|
disposed_at: str | None
|
||||||
|
disposition: dict[str, Any]
|
||||||
provenance: dict[str, Any]
|
provenance: dict[str, Any]
|
||||||
metadata: dict[str, Any]
|
metadata: dict[str, Any]
|
||||||
governance: DatasourceGovernancePayload
|
governance: DatasourceGovernancePayload
|
||||||
@@ -181,6 +186,7 @@ class DatasourceStageResponse(BaseModel):
|
|||||||
row_count: int | None
|
row_count: int | None
|
||||||
byte_count: int | None
|
byte_count: int | None
|
||||||
validation: DatasourceStageValidationResponse
|
validation: DatasourceStageValidationResponse
|
||||||
|
approval: dict[str, Any]
|
||||||
created_at: str | None
|
created_at: str | None
|
||||||
promoted_at: str | None
|
promoted_at: str | None
|
||||||
promoted_materialization_ref: str | None
|
promoted_materialization_ref: str | None
|
||||||
@@ -223,6 +229,61 @@ class DatasourceStagePromoteRequest(BaseModel):
|
|||||||
frozen_label: str | None = Field(default=None, max_length=300)
|
frozen_label: str | None = Field(default=None, max_length=300)
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceStageDecisionRequest(BaseModel):
|
||||||
|
decision: Literal["approve", "reject"]
|
||||||
|
reason: str = Field(min_length=1, max_length=2_000)
|
||||||
|
expected_policy_hash: str = Field(min_length=64, max_length=64)
|
||||||
|
expected_subject_digest: str = Field(min_length=64, max_length=64)
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceLifecycleEvidenceResponse(BaseModel):
|
||||||
|
ref: str
|
||||||
|
subject_ref: str
|
||||||
|
event_type: str
|
||||||
|
occurred_at: str
|
||||||
|
actor_ref: str | None
|
||||||
|
policy_version: str | None
|
||||||
|
policy_hash: str | None
|
||||||
|
subject_digest: str
|
||||||
|
previous_event_hash: str | None
|
||||||
|
event_hash: str
|
||||||
|
details: dict[str, Any]
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceLifecycleEvidenceListResponse(BaseModel):
|
||||||
|
evidence: list[DatasourceLifecycleEvidenceResponse]
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceRetentionCandidateResponse(BaseModel):
|
||||||
|
ref: str
|
||||||
|
kind: Literal["stage", "materialization"]
|
||||||
|
datasource_ref: str | None
|
||||||
|
disposition: Literal["delete_stage", "purge_materialization_payload"]
|
||||||
|
eligible_at: str
|
||||||
|
eligible: bool
|
||||||
|
blockers: list[str]
|
||||||
|
policy_version: str
|
||||||
|
policy_hash: str
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceRetentionPlanResponse(BaseModel):
|
||||||
|
as_of: str
|
||||||
|
plan_hash: str
|
||||||
|
candidates: list[DatasourceRetentionCandidateResponse]
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceRetentionApplyRequest(BaseModel):
|
||||||
|
as_of: datetime
|
||||||
|
plan_hash: str = Field(min_length=64, max_length=64)
|
||||||
|
target_refs: list[str] = Field(min_length=1, max_length=500)
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceRetentionApplyResponse(BaseModel):
|
||||||
|
plan_hash: str
|
||||||
|
disposed_refs: list[str]
|
||||||
|
evidence_hashes: list[str]
|
||||||
|
|
||||||
|
|
||||||
class DatasourceStagePromoteResponse(BaseModel):
|
class DatasourceStagePromoteResponse(BaseModel):
|
||||||
datasource: DatasourceResponse
|
datasource: DatasourceResponse
|
||||||
materialization: DatasourceMaterializationResponse
|
materialization: DatasourceMaterializationResponse
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import json
|
|||||||
import re
|
import re
|
||||||
from collections.abc import Mapping, Sequence
|
from collections.abc import Mapping, Sequence
|
||||||
from dataclasses import dataclass, replace
|
from dataclasses import dataclass, replace
|
||||||
|
from datetime import datetime
|
||||||
from typing import Any, cast
|
from typing import Any, cast
|
||||||
|
|
||||||
from sqlalchemy import exists, func, or_, select, text
|
from sqlalchemy import exists, func, or_, select, text
|
||||||
@@ -46,6 +47,18 @@ from govoplan_datasources.backend.db.models import (
|
|||||||
DatasourceRecord,
|
DatasourceRecord,
|
||||||
DatasourceStageRecord,
|
DatasourceStageRecord,
|
||||||
)
|
)
|
||||||
|
from govoplan_datasources.backend.governance import (
|
||||||
|
RetentionPlan,
|
||||||
|
apply_retention_plan,
|
||||||
|
build_retention_plan,
|
||||||
|
decide_stage,
|
||||||
|
ensure_stage_approval_current,
|
||||||
|
initialize_stage_approval,
|
||||||
|
list_lifecycle_evidence,
|
||||||
|
normalize_approval_policy,
|
||||||
|
normalize_retention_policy,
|
||||||
|
record_lifecycle_evidence,
|
||||||
|
)
|
||||||
from govoplan_datasources.backend.payloads import (
|
from govoplan_datasources.backend.payloads import (
|
||||||
DatasourcePayloadBackend,
|
DatasourcePayloadBackend,
|
||||||
PayloadBackendRegistry,
|
PayloadBackendRegistry,
|
||||||
@@ -77,6 +90,7 @@ from govoplan_datasources.backend.visibility import (
|
|||||||
CATALOGUE_READ_SCOPE = "datasources:catalogue:read"
|
CATALOGUE_READ_SCOPE = "datasources:catalogue:read"
|
||||||
SOURCE_WRITE_SCOPE = "datasources:source:write"
|
SOURCE_WRITE_SCOPE = "datasources:source:write"
|
||||||
STAGE_WRITE_SCOPE = "datasources:stage:write"
|
STAGE_WRITE_SCOPE = "datasources:stage:write"
|
||||||
|
STAGE_APPROVE_SCOPE = "datasources:stage:approve"
|
||||||
ADMIN_SCOPE = "datasources:source:admin"
|
ADMIN_SCOPE = "datasources:source:admin"
|
||||||
|
|
||||||
|
|
||||||
@@ -480,6 +494,10 @@ class SqlDatasourceProvider:
|
|||||||
else "This datasource has no materialized state."
|
else "This datasource has no materialized state."
|
||||||
)
|
)
|
||||||
raise DatasourceUnavailableError(message)
|
raise DatasourceUnavailableError(message)
|
||||||
|
if materialization.disposed_at is not None:
|
||||||
|
raise DatasourceUnavailableError(
|
||||||
|
"This materialization payload was disposed under its retention policy."
|
||||||
|
)
|
||||||
source_schema = _fields(materialization.schema_)
|
source_schema = _fields(materialization.schema_)
|
||||||
snapshot_governance = DatasourceGovernance.from_mapping(
|
snapshot_governance = DatasourceGovernance.from_mapping(
|
||||||
materialization.governance_snapshot_
|
materialization.governance_snapshot_
|
||||||
@@ -721,6 +739,8 @@ class SqlDatasourceProvider:
|
|||||||
governance.visibility_policy,
|
governance.visibility_policy,
|
||||||
schema=schema,
|
schema=schema,
|
||||||
),
|
),
|
||||||
|
approval_policy=normalize_approval_policy(governance.approval_policy),
|
||||||
|
retention_policy=normalize_retention_policy(governance.retention_policy),
|
||||||
)
|
)
|
||||||
fingerprint = fingerprint_rows(rows, schema)
|
fingerprint = fingerprint_rows(rows, schema)
|
||||||
validation = validate_stage(
|
validation = validate_stage(
|
||||||
@@ -738,7 +758,7 @@ class SqlDatasourceProvider:
|
|||||||
kind=stage.kind,
|
kind=stage.kind,
|
||||||
mode=stage.mode,
|
mode=stage.mode,
|
||||||
shape=stage.shape,
|
shape=stage.shape,
|
||||||
state="ready" if validation["valid"] else "invalid",
|
state="invalid",
|
||||||
provider=_clean_optional(stage.provider),
|
provider=_clean_optional(stage.provider),
|
||||||
provider_ref=_clean_optional(stage.provider_ref),
|
provider_ref=_clean_optional(stage.provider_ref),
|
||||||
schema_=[field_payload(field) for field in schema],
|
schema_=[field_payload(field) for field in schema],
|
||||||
@@ -754,8 +774,91 @@ class SqlDatasourceProvider:
|
|||||||
)
|
)
|
||||||
db.add(item)
|
db.add(item)
|
||||||
db.flush()
|
db.flush()
|
||||||
|
approval = initialize_stage_approval(item, created_at=item.created_at)
|
||||||
|
record_lifecycle_evidence(
|
||||||
|
db,
|
||||||
|
tenant_id=api_principal.tenant_id,
|
||||||
|
subject_ref=_stage_ref(item.id),
|
||||||
|
event_type="stage.validated",
|
||||||
|
actor_ref=_actor_id(api_principal),
|
||||||
|
subject_digest=str(approval["subject_digest"]),
|
||||||
|
policy_version=str(validation.get("policy_version") or "1"),
|
||||||
|
policy_hash=str(validation.get("policy_hash") or ""),
|
||||||
|
details={
|
||||||
|
"validation_valid": validation.get("valid"),
|
||||||
|
"schema_classification": (
|
||||||
|
validation.get("schema_change", {}).get("classification")
|
||||||
|
if isinstance(validation.get("schema_change"), Mapping)
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
"approval_required": approval.get("state") != "not_required",
|
||||||
|
"approval_policy_hash": approval.get("policy_hash"),
|
||||||
|
},
|
||||||
|
occurred_at=item.created_at,
|
||||||
|
)
|
||||||
|
db.flush()
|
||||||
return _stage_dto(item)
|
return _stage_dto(item)
|
||||||
|
|
||||||
|
def decide_stage(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
stage_ref: str,
|
||||||
|
decision: str,
|
||||||
|
reason: str,
|
||||||
|
expected_policy_hash: str,
|
||||||
|
expected_subject_digest: str,
|
||||||
|
) -> tuple[DatasourceStage, str | None, bool]:
|
||||||
|
db, api_principal = _context(session, principal, STAGE_APPROVE_SCOPE)
|
||||||
|
actor_ref = _actor_id(api_principal)
|
||||||
|
if actor_ref is None:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"An attributable account is required for datasource approval."
|
||||||
|
)
|
||||||
|
stage = _required_stage(
|
||||||
|
db,
|
||||||
|
tenant_id=api_principal.tenant_id,
|
||||||
|
stage_ref=stage_ref,
|
||||||
|
for_update=True,
|
||||||
|
)
|
||||||
|
approval, replayed = decide_stage(
|
||||||
|
stage,
|
||||||
|
actor_ref=actor_ref,
|
||||||
|
actor_scopes=tuple(api_principal.scopes),
|
||||||
|
decision=decision,
|
||||||
|
reason=reason,
|
||||||
|
expected_policy_hash=expected_policy_hash,
|
||||||
|
expected_subject_digest=expected_subject_digest,
|
||||||
|
)
|
||||||
|
evidence_hash = None
|
||||||
|
if not replayed:
|
||||||
|
policy = approval.get("policy")
|
||||||
|
policy_mapping = policy if isinstance(policy, Mapping) else {}
|
||||||
|
evidence = record_lifecycle_evidence(
|
||||||
|
db,
|
||||||
|
tenant_id=api_principal.tenant_id,
|
||||||
|
subject_ref=_stage_ref(stage.id),
|
||||||
|
event_type=(
|
||||||
|
"stage.approved" if decision == "approve" else "stage.rejected"
|
||||||
|
),
|
||||||
|
actor_ref=actor_ref,
|
||||||
|
subject_digest=str(approval["subject_digest"]),
|
||||||
|
policy_version=str(policy_mapping.get("version") or "1"),
|
||||||
|
policy_hash=str(approval["policy_hash"]),
|
||||||
|
details={
|
||||||
|
"decision": decision,
|
||||||
|
"reason": reason.strip(),
|
||||||
|
"approval_count": approval.get("approval_count"),
|
||||||
|
"required_approvals": approval.get("required_approvals"),
|
||||||
|
"resulting_state": approval.get("state"),
|
||||||
|
"authority_scopes": sorted(api_principal.scopes),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
evidence_hash = evidence.event_hash
|
||||||
|
db.flush()
|
||||||
|
return _stage_dto(stage), evidence_hash, replayed
|
||||||
|
|
||||||
def promote_stage(
|
def promote_stage(
|
||||||
self,
|
self,
|
||||||
session: object,
|
session: object,
|
||||||
@@ -787,6 +890,15 @@ class SqlDatasourceProvider:
|
|||||||
or datasource.deleted_at is not None
|
or datasource.deleted_at is not None
|
||||||
):
|
):
|
||||||
datasource = None
|
datasource = None
|
||||||
|
current_governance = (
|
||||||
|
_datasource_governance(datasource)
|
||||||
|
if datasource is not None
|
||||||
|
else DatasourceGovernance.from_mapping(stage.governance_)
|
||||||
|
)
|
||||||
|
ensure_stage_approval_current(
|
||||||
|
stage,
|
||||||
|
current_policy=current_governance.approval_policy,
|
||||||
|
)
|
||||||
if datasource is None:
|
if datasource is None:
|
||||||
_ensure_source_name_available(
|
_ensure_source_name_available(
|
||||||
db,
|
db,
|
||||||
@@ -845,6 +957,7 @@ class SqlDatasourceProvider:
|
|||||||
**dict(stage.provenance_),
|
**dict(stage.provenance_),
|
||||||
"stage_ref": _stage_ref(stage.id),
|
"stage_ref": _stage_ref(stage.id),
|
||||||
"stage_validation": dict(stage.validation_),
|
"stage_validation": dict(stage.validation_),
|
||||||
|
"stage_approval": dict(stage.approval_),
|
||||||
},
|
},
|
||||||
metadata=dict(stage.metadata_),
|
metadata=dict(stage.metadata_),
|
||||||
set_current=True,
|
set_current=True,
|
||||||
@@ -852,9 +965,81 @@ class SqlDatasourceProvider:
|
|||||||
stage.state = "promoted"
|
stage.state = "promoted"
|
||||||
stage.promoted_at = utcnow()
|
stage.promoted_at = utcnow()
|
||||||
stage.promoted_materialization_id = materialization.id
|
stage.promoted_materialization_id = materialization.id
|
||||||
|
approval_policy = stage.approval_.get("policy")
|
||||||
|
policy_mapping = (
|
||||||
|
approval_policy if isinstance(approval_policy, Mapping) else {}
|
||||||
|
)
|
||||||
|
promotion_evidence = record_lifecycle_evidence(
|
||||||
|
db,
|
||||||
|
tenant_id=api_principal.tenant_id,
|
||||||
|
subject_ref=_stage_ref(stage.id),
|
||||||
|
event_type="stage.promoted",
|
||||||
|
actor_ref=_actor_id(api_principal),
|
||||||
|
subject_digest=str(stage.approval_.get("subject_digest") or ""),
|
||||||
|
policy_version=str(policy_mapping.get("version") or "1"),
|
||||||
|
policy_hash=str(stage.approval_.get("policy_hash") or ""),
|
||||||
|
details={
|
||||||
|
"datasource_ref": _datasource_ref(datasource.id),
|
||||||
|
"materialization_ref": _materialization_ref(materialization.id),
|
||||||
|
"revision": materialization.revision,
|
||||||
|
"quality_policy_hash": stage.validation_.get("policy_hash"),
|
||||||
|
"approval_count": stage.approval_.get("approval_count"),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
materialization.provenance_ = {
|
||||||
|
**dict(materialization.provenance_),
|
||||||
|
"promotion_evidence_hash": promotion_evidence.event_hash,
|
||||||
|
}
|
||||||
db.flush()
|
db.flush()
|
||||||
return _datasource_dto(datasource), _materialization_dto(materialization)
|
return _datasource_dto(datasource), _materialization_dto(materialization)
|
||||||
|
|
||||||
|
def prepare_refresh(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
datasource_ref: str,
|
||||||
|
) -> DatasourceStage:
|
||||||
|
db, api_principal = _context(session, principal, SOURCE_WRITE_SCOPE)
|
||||||
|
item = _required_datasource(
|
||||||
|
db,
|
||||||
|
tenant_id=api_principal.tenant_id,
|
||||||
|
datasource_ref=datasource_ref,
|
||||||
|
)
|
||||||
|
if item.mode != "cached" or not item.provider_ref:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"Only cached connector-backed datasources can prepare a refresh."
|
||||||
|
)
|
||||||
|
rows, origin = self._read_origin_all(
|
||||||
|
db,
|
||||||
|
api_principal,
|
||||||
|
origin_ref=item.provider_ref,
|
||||||
|
)
|
||||||
|
return self.create_stage(
|
||||||
|
db,
|
||||||
|
api_principal,
|
||||||
|
stage=DatasourceStageInput(
|
||||||
|
name=item.name,
|
||||||
|
source_name=item.source_name,
|
||||||
|
description=item.description,
|
||||||
|
kind=cast(Any, item.kind),
|
||||||
|
mode="cached",
|
||||||
|
shape=cast(Any, item.shape),
|
||||||
|
rows=tuple(rows),
|
||||||
|
target_datasource_ref=_datasource_ref(item.id),
|
||||||
|
provider=item.provider,
|
||||||
|
provider_ref=item.provider_ref,
|
||||||
|
provenance={
|
||||||
|
"created_via": "datasources.refresh",
|
||||||
|
"origin_ref": origin.ref,
|
||||||
|
"origin_fingerprint": origin.fingerprint,
|
||||||
|
"origin_schema_version": origin.schema_version,
|
||||||
|
},
|
||||||
|
metadata=dict(item.metadata_),
|
||||||
|
governance=_datasource_governance(item),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
def register_origin(
|
def register_origin(
|
||||||
self,
|
self,
|
||||||
session: object,
|
session: object,
|
||||||
@@ -973,6 +1158,13 @@ class SqlDatasourceProvider:
|
|||||||
raise DatasourceValidationError(
|
raise DatasourceValidationError(
|
||||||
"Only cached connector-backed datasources can be refreshed."
|
"Only cached connector-backed datasources can be refreshed."
|
||||||
)
|
)
|
||||||
|
approval_policy = normalize_approval_policy(
|
||||||
|
_datasource_governance(item).approval_policy
|
||||||
|
)
|
||||||
|
if approval_policy["required"]:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"This datasource requires an approved refresh stage before it can become current."
|
||||||
|
)
|
||||||
rows, origin = self._read_origin_all(
|
rows, origin = self._read_origin_all(
|
||||||
db,
|
db,
|
||||||
api_principal,
|
api_principal,
|
||||||
@@ -988,6 +1180,68 @@ class SqlDatasourceProvider:
|
|||||||
)
|
)
|
||||||
return _datasource_dto(item), _materialization_dto(materialization)
|
return _datasource_dto(item), _materialization_dto(materialization)
|
||||||
|
|
||||||
|
def preview_retention(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
as_of: datetime,
|
||||||
|
) -> RetentionPlan:
|
||||||
|
db, api_principal = _context(session, principal, ADMIN_SCOPE)
|
||||||
|
return build_retention_plan(
|
||||||
|
db,
|
||||||
|
tenant_id=api_principal.tenant_id,
|
||||||
|
as_of=as_of,
|
||||||
|
)
|
||||||
|
|
||||||
|
def apply_retention(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
as_of: datetime,
|
||||||
|
plan_hash: str,
|
||||||
|
target_refs: Sequence[str],
|
||||||
|
) -> tuple[tuple[str, ...], tuple[str, ...]]:
|
||||||
|
db, api_principal = _context(session, principal, ADMIN_SCOPE)
|
||||||
|
actor_ref = _actor_id(api_principal)
|
||||||
|
if actor_ref is None:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"An attributable account is required for retention execution."
|
||||||
|
)
|
||||||
|
plan = build_retention_plan(
|
||||||
|
db,
|
||||||
|
tenant_id=api_principal.tenant_id,
|
||||||
|
as_of=as_of,
|
||||||
|
)
|
||||||
|
if plan.plan_hash != plan_hash:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"The retention plan changed; preview it again before applying."
|
||||||
|
)
|
||||||
|
return apply_retention_plan(
|
||||||
|
db,
|
||||||
|
tenant_id=api_principal.tenant_id,
|
||||||
|
actor_ref=actor_ref,
|
||||||
|
plan=plan,
|
||||||
|
target_refs=target_refs,
|
||||||
|
)
|
||||||
|
|
||||||
|
def list_lifecycle_evidence(
|
||||||
|
self,
|
||||||
|
session: object,
|
||||||
|
principal: object,
|
||||||
|
*,
|
||||||
|
subject_ref: str | None = None,
|
||||||
|
limit: int = 200,
|
||||||
|
):
|
||||||
|
db, api_principal = _context(session, principal, CATALOGUE_READ_SCOPE)
|
||||||
|
return list_lifecycle_evidence(
|
||||||
|
db,
|
||||||
|
tenant_id=api_principal.tenant_id,
|
||||||
|
subject_ref=subject_ref,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
|
||||||
def freeze_datasource(
|
def freeze_datasource(
|
||||||
self,
|
self,
|
||||||
session: object,
|
session: object,
|
||||||
@@ -1741,16 +1995,18 @@ def _required_stage(
|
|||||||
*,
|
*,
|
||||||
tenant_id: str,
|
tenant_id: str,
|
||||||
stage_ref: str,
|
stage_ref: str,
|
||||||
|
for_update: bool = False,
|
||||||
) -> DatasourceStageRecord:
|
) -> DatasourceStageRecord:
|
||||||
stage_id = _strip_ref(stage_ref, "stage:")
|
stage_id = _strip_ref(stage_ref, "stage:")
|
||||||
if stage_id is None:
|
if stage_id is None:
|
||||||
raise DatasourceNotFoundError("Datasource stage not found.")
|
raise DatasourceNotFoundError("Datasource stage not found.")
|
||||||
item = session.scalar(
|
statement = select(DatasourceStageRecord).where(
|
||||||
select(DatasourceStageRecord).where(
|
|
||||||
DatasourceStageRecord.id == stage_id,
|
DatasourceStageRecord.id == stage_id,
|
||||||
DatasourceStageRecord.tenant_id == tenant_id,
|
DatasourceStageRecord.tenant_id == tenant_id,
|
||||||
)
|
)
|
||||||
)
|
if for_update:
|
||||||
|
statement = statement.with_for_update()
|
||||||
|
item = session.scalar(statement)
|
||||||
if item is None:
|
if item is None:
|
||||||
raise DatasourceNotFoundError("Datasource stage not found.")
|
raise DatasourceNotFoundError("Datasource stage not found.")
|
||||||
return item
|
return item
|
||||||
@@ -1817,6 +2073,8 @@ def _datasource_governance(item: DatasourceRecord) -> DatasourceGovernance:
|
|||||||
"transfer_agreement_ref": item.transfer_agreement_ref,
|
"transfer_agreement_ref": item.transfer_agreement_ref,
|
||||||
"freshness_policy": item.freshness_policy,
|
"freshness_policy": item.freshness_policy,
|
||||||
"quality_policy": item.quality_policy,
|
"quality_policy": item.quality_policy,
|
||||||
|
"approval_policy": item.approval_policy,
|
||||||
|
"retention_policy": item.retention_policy,
|
||||||
"known_limits": item.known_limits,
|
"known_limits": item.known_limits,
|
||||||
"correction_procedure_ref": item.correction_procedure_ref,
|
"correction_procedure_ref": item.correction_procedure_ref,
|
||||||
"affected_refs": item.affected_refs,
|
"affected_refs": item.affected_refs,
|
||||||
@@ -1855,6 +2113,8 @@ def _apply_datasource_governance(
|
|||||||
item.transfer_agreement_ref = governance.transfer_agreement_ref
|
item.transfer_agreement_ref = governance.transfer_agreement_ref
|
||||||
item.freshness_policy = dict(governance.freshness_policy)
|
item.freshness_policy = dict(governance.freshness_policy)
|
||||||
item.quality_policy = dict(governance.quality_policy)
|
item.quality_policy = dict(governance.quality_policy)
|
||||||
|
item.approval_policy = normalize_approval_policy(governance.approval_policy)
|
||||||
|
item.retention_policy = normalize_retention_policy(governance.retention_policy)
|
||||||
item.known_limits = list(governance.known_limits)
|
item.known_limits = list(governance.known_limits)
|
||||||
item.correction_procedure_ref = governance.correction_procedure_ref
|
item.correction_procedure_ref = governance.correction_procedure_ref
|
||||||
item.affected_refs = list(governance.affected_refs)
|
item.affected_refs = list(governance.affected_refs)
|
||||||
@@ -1922,6 +2182,8 @@ def _materialization_dto(
|
|||||||
frozen_label=item.frozen_label,
|
frozen_label=item.frozen_label,
|
||||||
source_timestamp=item.source_timestamp,
|
source_timestamp=item.source_timestamp,
|
||||||
created_at=item.created_at,
|
created_at=item.created_at,
|
||||||
|
disposed_at=item.disposed_at,
|
||||||
|
disposition=dict(item.disposition_),
|
||||||
provenance=dict(item.provenance_),
|
provenance=dict(item.provenance_),
|
||||||
metadata=dict(item.metadata_),
|
metadata=dict(item.metadata_),
|
||||||
governance=DatasourceGovernance.from_mapping(item.governance_snapshot_),
|
governance=DatasourceGovernance.from_mapping(item.governance_snapshot_),
|
||||||
@@ -1947,6 +2209,7 @@ def _stage_dto(item: DatasourceStageRecord) -> DatasourceStage:
|
|||||||
row_count=item.row_count,
|
row_count=item.row_count,
|
||||||
byte_count=item.byte_count,
|
byte_count=item.byte_count,
|
||||||
validation=dict(item.validation_),
|
validation=dict(item.validation_),
|
||||||
|
approval=dict(item.approval_),
|
||||||
created_at=item.created_at,
|
created_at=item.created_at,
|
||||||
promoted_at=item.promoted_at,
|
promoted_at=item.promoted_at,
|
||||||
promoted_materialization_ref=(
|
promoted_materialization_ref=(
|
||||||
@@ -2689,6 +2952,7 @@ __all__ = [
|
|||||||
"ADMIN_SCOPE",
|
"ADMIN_SCOPE",
|
||||||
"CATALOGUE_READ_SCOPE",
|
"CATALOGUE_READ_SCOPE",
|
||||||
"SOURCE_WRITE_SCOPE",
|
"SOURCE_WRITE_SCOPE",
|
||||||
|
"STAGE_APPROVE_SCOPE",
|
||||||
"STAGE_WRITE_SCOPE",
|
"STAGE_WRITE_SCOPE",
|
||||||
"SqlDatasourceProvider",
|
"SqlDatasourceProvider",
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ from govoplan_core.privacy.dsar_workflow import (
|
|||||||
)
|
)
|
||||||
from govoplan_datasources.backend.db.models import (
|
from govoplan_datasources.backend.db.models import (
|
||||||
DatasourceGovernanceReferenceRecord,
|
DatasourceGovernanceReferenceRecord,
|
||||||
|
DatasourceLifecycleEvidenceRecord,
|
||||||
DatasourceMaterializationRecord,
|
DatasourceMaterializationRecord,
|
||||||
DatasourcePayloadRecord,
|
DatasourcePayloadRecord,
|
||||||
DatasourcePayloadRowRecord,
|
DatasourcePayloadRowRecord,
|
||||||
@@ -382,6 +383,54 @@ class DatasourcesDsarProviderTests(unittest.TestCase):
|
|||||||
self.assertEqual((), wrong_tenant)
|
self.assertEqual((), wrong_tenant)
|
||||||
self.assertEqual((), conflict)
|
self.assertEqual((), conflict)
|
||||||
|
|
||||||
|
def test_lifecycle_evidence_is_minimized_and_immutable(self) -> None:
|
||||||
|
evidence = DatasourceLifecycleEvidenceRecord(
|
||||||
|
id="evidence-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
subject_ref="datasource:datasource-1",
|
||||||
|
event_type="retention.applied",
|
||||||
|
occurred_at=NOW,
|
||||||
|
actor_ref="account-1",
|
||||||
|
policy_version="retention-v1",
|
||||||
|
policy_hash="1" * 64,
|
||||||
|
subject_digest="2" * 64,
|
||||||
|
event_hash="3" * 64,
|
||||||
|
details_={"secret": SECRET},
|
||||||
|
)
|
||||||
|
self.session.add(evidence)
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
exact_subject = DsarSubjectRef(
|
||||||
|
external_references={
|
||||||
|
"datasources.lifecycle_evidence": "evidence-1",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
exact = self.provider.search_subject(
|
||||||
|
self.session,
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
subject=exact_subject,
|
||||||
|
)
|
||||||
|
self.assertEqual(1, len(exact))
|
||||||
|
self.assertEqual("datasource_operator_attribution", exact[0].category)
|
||||||
|
self.assertNotIn(SECRET, json.dumps(exact[0].to_dict()))
|
||||||
|
actions = self.provider.plan_erasure(
|
||||||
|
self.session,
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
subject=exact_subject,
|
||||||
|
records=exact,
|
||||||
|
)
|
||||||
|
self.assertEqual({"retain"}, {action.kind for action in actions})
|
||||||
|
|
||||||
|
canonical = self.provider.search_subject(
|
||||||
|
self.session,
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
subject=DsarSubjectRef(account_id="account-1"),
|
||||||
|
)
|
||||||
|
self.assertIn(
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
{record.resource_type for record in canonical},
|
||||||
|
)
|
||||||
|
|
||||||
def test_transient_deletion_is_safe_and_idempotent(self) -> None:
|
def test_transient_deletion_is_safe_and_idempotent(self) -> None:
|
||||||
subject = DsarSubjectRef(
|
subject = DsarSubjectRef(
|
||||||
external_references={
|
external_references={
|
||||||
|
|||||||
@@ -18,6 +18,8 @@ class DatasourcesInterfaceDocumentationContractTests(unittest.TestCase):
|
|||||||
"datasources.origins",
|
"datasources.origins",
|
||||||
"datasources.governance",
|
"datasources.governance",
|
||||||
"datasources.preview",
|
"datasources.preview",
|
||||||
|
"datasources.lifecycle-evidence",
|
||||||
|
"datasources.retention",
|
||||||
},
|
},
|
||||||
{item.id for item in frontend.view_surfaces}, # type: ignore[union-attr]
|
{item.id for item in frontend.view_surfaces}, # type: ignore[union-attr]
|
||||||
)
|
)
|
||||||
@@ -29,6 +31,7 @@ class DatasourcesInterfaceDocumentationContractTests(unittest.TestCase):
|
|||||||
quality = topics["datasources.quality-gates"]
|
quality = topics["datasources.quality-gates"]
|
||||||
visibility = topics["datasources.visibility"]
|
visibility = topics["datasources.visibility"]
|
||||||
reference = topics["datasources.reference.fields-and-consequences"]
|
reference = topics["datasources.reference.fields-and-consequences"]
|
||||||
|
lifecycle_controls = topics["datasources.approval-and-retention"]
|
||||||
|
|
||||||
self.assertIn("datasources.staging", lifecycle.metadata["help_contexts"])
|
self.assertIn("datasources.staging", lifecycle.metadata["help_contexts"])
|
||||||
self.assertIn("datasources.field.authority-mode", governance.metadata["help_contexts"])
|
self.assertIn("datasources.field.authority-mode", governance.metadata["help_contexts"])
|
||||||
@@ -43,6 +46,9 @@ class DatasourcesInterfaceDocumentationContractTests(unittest.TestCase):
|
|||||||
self.assertIn("datasources.action.promote", reference.metadata["help_contexts"])
|
self.assertIn("datasources.action.promote", reference.metadata["help_contexts"])
|
||||||
self.assertIn("freeze", reference.metadata["consequence_classes"])
|
self.assertIn("freeze", reference.metadata["consequence_classes"])
|
||||||
self.assertIn("retire", reference.metadata["consequence_classes"])
|
self.assertIn("retire", reference.metadata["consequence_classes"])
|
||||||
|
self.assertIn("datasources.action.approve", lifecycle_controls.metadata["help_contexts"])
|
||||||
|
self.assertIn("hash-chained lifecycle evidence", lifecycle_controls.body)
|
||||||
|
self.assertTrue(lifecycle_controls.translations["de"]["body"])
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -31,6 +31,7 @@ from govoplan_core.core.tabular_sources import (
|
|||||||
from govoplan_core.db.base import Base, utcnow
|
from govoplan_core.db.base import Base, utcnow
|
||||||
from govoplan_datasources.backend.db.models import (
|
from govoplan_datasources.backend.db.models import (
|
||||||
DatasourceGovernanceReferenceRecord,
|
DatasourceGovernanceReferenceRecord,
|
||||||
|
DatasourceLifecycleEvidenceRecord,
|
||||||
DatasourceMaterializationRecord,
|
DatasourceMaterializationRecord,
|
||||||
DatasourcePayloadRecord,
|
DatasourcePayloadRecord,
|
||||||
DatasourcePayloadRowRecord,
|
DatasourcePayloadRowRecord,
|
||||||
@@ -226,6 +227,7 @@ class DatasourceLifecycleTests(unittest.TestCase):
|
|||||||
tables=[
|
tables=[
|
||||||
DatasourceRecord.__table__,
|
DatasourceRecord.__table__,
|
||||||
DatasourceGovernanceReferenceRecord.__table__,
|
DatasourceGovernanceReferenceRecord.__table__,
|
||||||
|
DatasourceLifecycleEvidenceRecord.__table__,
|
||||||
DatasourcePayloadRecord.__table__,
|
DatasourcePayloadRecord.__table__,
|
||||||
DatasourcePayloadRowRecord.__table__,
|
DatasourcePayloadRowRecord.__table__,
|
||||||
DatasourceMaterializationRecord.__table__,
|
DatasourceMaterializationRecord.__table__,
|
||||||
@@ -248,6 +250,7 @@ class DatasourceLifecycleTests(unittest.TestCase):
|
|||||||
Base.metadata.drop_all(
|
Base.metadata.drop_all(
|
||||||
self.engine,
|
self.engine,
|
||||||
tables=[
|
tables=[
|
||||||
|
DatasourceLifecycleEvidenceRecord.__table__,
|
||||||
DatasourceStageRecord.__table__,
|
DatasourceStageRecord.__table__,
|
||||||
DatasourcePublicationRecord.__table__,
|
DatasourcePublicationRecord.__table__,
|
||||||
DatasourceMaterializationRecord.__table__,
|
DatasourceMaterializationRecord.__table__,
|
||||||
|
|||||||
@@ -0,0 +1,497 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
from datetime import timedelta
|
||||||
|
|
||||||
|
from sqlalchemy import create_engine, select
|
||||||
|
from sqlalchemy.orm import sessionmaker
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal
|
||||||
|
from govoplan_core.core.access import PrincipalRef
|
||||||
|
from govoplan_core.core.datasources import (
|
||||||
|
CAPABILITY_DATASOURCE_ORIGINS,
|
||||||
|
DatasourceField,
|
||||||
|
DatasourceGovernance,
|
||||||
|
DatasourceOrigin,
|
||||||
|
DatasourceOriginReadRequest,
|
||||||
|
DatasourceOriginReadResult,
|
||||||
|
DatasourceReadRequest,
|
||||||
|
DatasourceStageInput,
|
||||||
|
DatasourceUnavailableError,
|
||||||
|
DatasourceValidationError,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.tabular_sources import TabularPushdown, TabularSourceHealth
|
||||||
|
from govoplan_core.db.base import Base
|
||||||
|
from govoplan_datasources.backend.db.models import (
|
||||||
|
DatasourceGovernanceReferenceRecord,
|
||||||
|
DatasourceLifecycleEvidenceRecord,
|
||||||
|
DatasourceMaterializationRecord,
|
||||||
|
DatasourcePayloadRecord,
|
||||||
|
DatasourcePayloadRowRecord,
|
||||||
|
DatasourcePublicationRecord,
|
||||||
|
DatasourceRecord,
|
||||||
|
DatasourceStageRecord,
|
||||||
|
)
|
||||||
|
from govoplan_datasources.backend.service import (
|
||||||
|
CATALOGUE_READ_SCOPE,
|
||||||
|
SOURCE_WRITE_SCOPE,
|
||||||
|
STAGE_APPROVE_SCOPE,
|
||||||
|
STAGE_WRITE_SCOPE,
|
||||||
|
SqlDatasourceProvider,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def principal(
|
||||||
|
account_id: str,
|
||||||
|
*,
|
||||||
|
scopes: tuple[str, ...],
|
||||||
|
) -> ApiPrincipal:
|
||||||
|
return ApiPrincipal(
|
||||||
|
principal=PrincipalRef(
|
||||||
|
account_id=account_id,
|
||||||
|
membership_id=f"membership-{account_id}",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
scopes=frozenset(scopes),
|
||||||
|
),
|
||||||
|
account=object(),
|
||||||
|
user=object(),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class _OriginProvider:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.rows = [{"id": 1, "status": "initial"}]
|
||||||
|
|
||||||
|
def origin(self) -> DatasourceOrigin:
|
||||||
|
return DatasourceOrigin(
|
||||||
|
ref="origin:cases",
|
||||||
|
source_name="cached_cases",
|
||||||
|
name="Cached cases",
|
||||||
|
kind="database",
|
||||||
|
shape="tabular",
|
||||||
|
supported_modes=("cached",),
|
||||||
|
provider="connectors.test",
|
||||||
|
schema=(
|
||||||
|
DatasourceField("id", "integer", False),
|
||||||
|
DatasourceField("status", "string", False),
|
||||||
|
),
|
||||||
|
schema_version="1",
|
||||||
|
fingerprint=f"cases-{self.rows[-1]['status']}",
|
||||||
|
row_count=len(self.rows),
|
||||||
|
source_mode="cached",
|
||||||
|
pushdown=TabularPushdown(pagination=True),
|
||||||
|
health=TabularSourceHealth(
|
||||||
|
status="healthy",
|
||||||
|
code="origin.ready",
|
||||||
|
summary="Origin is ready.",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def list_origins(self, _session, _principal, *, query="", limit=100):
|
||||||
|
del limit
|
||||||
|
origin = self.origin()
|
||||||
|
return (origin,) if query.casefold() in origin.name.casefold() else ()
|
||||||
|
|
||||||
|
def get_origin(self, _session, _principal, *, origin_ref):
|
||||||
|
return self.origin() if origin_ref == "origin:cases" else None
|
||||||
|
|
||||||
|
def read_origin(
|
||||||
|
self,
|
||||||
|
_session,
|
||||||
|
_principal,
|
||||||
|
*,
|
||||||
|
request: DatasourceOriginReadRequest,
|
||||||
|
) -> DatasourceOriginReadResult:
|
||||||
|
rows = self.rows[request.offset : request.offset + request.limit]
|
||||||
|
return DatasourceOriginReadResult(
|
||||||
|
origin=self.origin(),
|
||||||
|
rows=tuple(dict(row) for row in rows),
|
||||||
|
total_rows=len(self.rows),
|
||||||
|
truncated=request.offset + len(rows) < len(self.rows),
|
||||||
|
returned_bytes=64,
|
||||||
|
elapsed_ms=1,
|
||||||
|
effective_row_limit=request.limit,
|
||||||
|
effective_byte_limit=request.max_bytes,
|
||||||
|
effective_timeout_ms=request.timeout_ms,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class _Registry:
|
||||||
|
def __init__(self, origins: _OriginProvider) -> None:
|
||||||
|
self.origins = origins
|
||||||
|
|
||||||
|
def has_capability(self, name: str) -> bool:
|
||||||
|
return name == CAPABILITY_DATASOURCE_ORIGINS
|
||||||
|
|
||||||
|
def capability(self, name: str):
|
||||||
|
if not self.has_capability(name):
|
||||||
|
raise KeyError(name)
|
||||||
|
return self.origins
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceLifecycleGovernanceTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.engine = create_engine("sqlite:///:memory:")
|
||||||
|
self.tables = [
|
||||||
|
DatasourceRecord.__table__,
|
||||||
|
DatasourceGovernanceReferenceRecord.__table__,
|
||||||
|
DatasourcePayloadRecord.__table__,
|
||||||
|
DatasourcePayloadRowRecord.__table__,
|
||||||
|
DatasourceMaterializationRecord.__table__,
|
||||||
|
DatasourceStageRecord.__table__,
|
||||||
|
DatasourcePublicationRecord.__table__,
|
||||||
|
DatasourceLifecycleEvidenceRecord.__table__,
|
||||||
|
]
|
||||||
|
Base.metadata.create_all(self.engine, tables=self.tables)
|
||||||
|
self.session = sessionmaker(bind=self.engine)()
|
||||||
|
self.origins = _OriginProvider()
|
||||||
|
self.provider = SqlDatasourceProvider(registry=_Registry(self.origins))
|
||||||
|
self.writer = principal(
|
||||||
|
"writer",
|
||||||
|
scopes=(
|
||||||
|
CATALOGUE_READ_SCOPE,
|
||||||
|
SOURCE_WRITE_SCOPE,
|
||||||
|
STAGE_WRITE_SCOPE,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def tearDown(self) -> None:
|
||||||
|
self.session.close()
|
||||||
|
Base.metadata.drop_all(self.engine, tables=list(reversed(self.tables)))
|
||||||
|
self.engine.dispose()
|
||||||
|
|
||||||
|
def test_approval_quorum_is_attributable_idempotent_and_separated(self) -> None:
|
||||||
|
stage = self.provider.create_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage=DatasourceStageInput(
|
||||||
|
name="Approval register",
|
||||||
|
source_name="approval_register",
|
||||||
|
kind="upload",
|
||||||
|
mode="static",
|
||||||
|
shape="tabular",
|
||||||
|
rows=({"id": 1},),
|
||||||
|
governance=DatasourceGovernance(
|
||||||
|
approval_policy={
|
||||||
|
"version": "promotion-v2",
|
||||||
|
"required": True,
|
||||||
|
"required_approvals": 2,
|
||||||
|
"separation_of_duties": True,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self.assertEqual("awaiting_approval", stage.state)
|
||||||
|
policy_hash = str(stage.approval["policy_hash"])
|
||||||
|
subject_digest = str(stage.approval["subject_digest"])
|
||||||
|
|
||||||
|
creator_approver = principal(
|
||||||
|
"writer",
|
||||||
|
scopes=(STAGE_APPROVE_SCOPE,),
|
||||||
|
)
|
||||||
|
with self.assertRaisesRegex(
|
||||||
|
DatasourceValidationError,
|
||||||
|
"creator cannot approve",
|
||||||
|
):
|
||||||
|
self.provider.decide_stage(
|
||||||
|
self.session,
|
||||||
|
creator_approver,
|
||||||
|
stage_ref=stage.ref,
|
||||||
|
decision="approve",
|
||||||
|
reason="Creator review",
|
||||||
|
expected_policy_hash=policy_hash,
|
||||||
|
expected_subject_digest=subject_digest,
|
||||||
|
)
|
||||||
|
|
||||||
|
first, first_hash, replayed = self.provider.decide_stage(
|
||||||
|
self.session,
|
||||||
|
principal("approver-1", scopes=(STAGE_APPROVE_SCOPE,)),
|
||||||
|
stage_ref=stage.ref,
|
||||||
|
decision="approve",
|
||||||
|
reason="Quality and schema evidence reviewed.",
|
||||||
|
expected_policy_hash=policy_hash,
|
||||||
|
expected_subject_digest=subject_digest,
|
||||||
|
)
|
||||||
|
self.assertFalse(replayed)
|
||||||
|
self.assertTrue(first_hash)
|
||||||
|
self.assertEqual("awaiting_approval", first.state)
|
||||||
|
|
||||||
|
replay, replay_hash, replayed = self.provider.decide_stage(
|
||||||
|
self.session,
|
||||||
|
principal("approver-1", scopes=(STAGE_APPROVE_SCOPE,)),
|
||||||
|
stage_ref=stage.ref,
|
||||||
|
decision="approve",
|
||||||
|
reason="Quality and schema evidence reviewed.",
|
||||||
|
expected_policy_hash=policy_hash,
|
||||||
|
expected_subject_digest=subject_digest,
|
||||||
|
)
|
||||||
|
self.assertTrue(replayed)
|
||||||
|
self.assertIsNone(replay_hash)
|
||||||
|
self.assertEqual(1, replay.approval["approval_count"])
|
||||||
|
|
||||||
|
approved, second_hash, replayed = self.provider.decide_stage(
|
||||||
|
self.session,
|
||||||
|
principal("approver-2", scopes=(STAGE_APPROVE_SCOPE,)),
|
||||||
|
stage_ref=stage.ref,
|
||||||
|
decision="approve",
|
||||||
|
reason="Authority and promotion impact reviewed.",
|
||||||
|
expected_policy_hash=policy_hash,
|
||||||
|
expected_subject_digest=subject_digest,
|
||||||
|
)
|
||||||
|
self.assertFalse(replayed)
|
||||||
|
self.assertTrue(second_hash)
|
||||||
|
self.assertEqual("ready", approved.state)
|
||||||
|
|
||||||
|
datasource, materialization = self.provider.promote_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage_ref=stage.ref,
|
||||||
|
)
|
||||||
|
self.assertEqual(2, materialization.provenance["stage_approval"]["approval_count"])
|
||||||
|
self.assertTrue(materialization.provenance["promotion_evidence_hash"])
|
||||||
|
self.assertEqual(materialization.ref, datasource.current_materialization_ref)
|
||||||
|
|
||||||
|
evidence = tuple(
|
||||||
|
reversed(
|
||||||
|
self.provider.list_lifecycle_evidence(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
subject_ref=stage.ref,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
["stage.validated", "stage.approved", "stage.approved", "stage.promoted"],
|
||||||
|
[item.event_type for item in evidence],
|
||||||
|
)
|
||||||
|
for previous, current in zip(evidence, evidence[1:], strict=False):
|
||||||
|
self.assertEqual(previous.event_hash, current.previous_event_hash)
|
||||||
|
|
||||||
|
def test_approved_refresh_stage_is_required_before_current_changes(self) -> None:
|
||||||
|
datasource = self.provider.register_origin(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
origin_ref="origin:cases",
|
||||||
|
name="Cached cases",
|
||||||
|
source_name="cached_cases",
|
||||||
|
mode="cached",
|
||||||
|
governance=DatasourceGovernance(
|
||||||
|
approval_policy={
|
||||||
|
"version": "refresh-v1",
|
||||||
|
"required": True,
|
||||||
|
"required_approvals": 1,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
)
|
||||||
|
original_ref = datasource.current_materialization_ref
|
||||||
|
self.origins.rows = [{"id": 1, "status": "refreshed"}]
|
||||||
|
|
||||||
|
with self.assertRaisesRegex(
|
||||||
|
DatasourceValidationError,
|
||||||
|
"approved refresh stage",
|
||||||
|
):
|
||||||
|
self.provider.refresh_datasource(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
datasource_ref=datasource.ref,
|
||||||
|
)
|
||||||
|
|
||||||
|
stage = self.provider.prepare_refresh(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
datasource_ref=datasource.ref,
|
||||||
|
)
|
||||||
|
self.assertEqual("awaiting_approval", stage.state)
|
||||||
|
approved, _, _ = self.provider.decide_stage(
|
||||||
|
self.session,
|
||||||
|
principal("refresh-approver", scopes=(STAGE_APPROVE_SCOPE,)),
|
||||||
|
stage_ref=stage.ref,
|
||||||
|
decision="approve",
|
||||||
|
reason="The connector delta and schema are acceptable.",
|
||||||
|
expected_policy_hash=str(stage.approval["policy_hash"]),
|
||||||
|
expected_subject_digest=str(stage.approval["subject_digest"]),
|
||||||
|
)
|
||||||
|
self.assertEqual("ready", approved.state)
|
||||||
|
refreshed, current = self.provider.promote_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage_ref=stage.ref,
|
||||||
|
)
|
||||||
|
self.assertNotEqual(original_ref, refreshed.current_materialization_ref)
|
||||||
|
self.assertEqual(current.ref, refreshed.current_materialization_ref)
|
||||||
|
preview = self.provider.read_datasource(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
request=DatasourceReadRequest(datasource_ref=datasource.ref),
|
||||||
|
)
|
||||||
|
self.assertEqual("refreshed", preview.rows[0]["status"])
|
||||||
|
|
||||||
|
def test_retention_preview_blocks_current_and_held_evidence_then_purges_payload(self) -> None:
|
||||||
|
governance = DatasourceGovernance(
|
||||||
|
retention_policy={
|
||||||
|
"version": "records-v3",
|
||||||
|
"enabled": True,
|
||||||
|
"materialization_days": 1,
|
||||||
|
"frozen_evidence_days": 1,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
first_stage = self.provider.create_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage=DatasourceStageInput(
|
||||||
|
name="Retained register",
|
||||||
|
source_name="retained_register",
|
||||||
|
kind="upload",
|
||||||
|
mode="static",
|
||||||
|
shape="tabular",
|
||||||
|
rows=({"id": 1},),
|
||||||
|
governance=governance,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
datasource, first = self.provider.promote_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage_ref=first_stage.ref,
|
||||||
|
)
|
||||||
|
second_stage = self.provider.create_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage=DatasourceStageInput(
|
||||||
|
name="Retained register",
|
||||||
|
source_name="retained_register",
|
||||||
|
kind="upload",
|
||||||
|
mode="static",
|
||||||
|
shape="tabular",
|
||||||
|
rows=({"id": 2},),
|
||||||
|
target_datasource_ref=datasource.ref,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
_, second = self.provider.promote_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage_ref=second_stage.ref,
|
||||||
|
)
|
||||||
|
as_of = max(first.created_at, second.created_at) + timedelta(days=2)
|
||||||
|
plan = self.provider.preview_retention(
|
||||||
|
self.session,
|
||||||
|
principal("admin", scopes=("datasources:source:admin",)),
|
||||||
|
as_of=as_of,
|
||||||
|
)
|
||||||
|
by_ref = {item.ref: item for item in plan.candidates}
|
||||||
|
self.assertTrue(by_ref[first.ref].eligible)
|
||||||
|
self.assertFalse(by_ref[second.ref].eligible)
|
||||||
|
self.assertIn("current_materialization", by_ref[second.ref].blockers)
|
||||||
|
|
||||||
|
first_row = self.session.scalar(
|
||||||
|
select(DatasourceMaterializationRecord).where(
|
||||||
|
DatasourceMaterializationRecord.id
|
||||||
|
== first.ref.removeprefix("materialization:")
|
||||||
|
)
|
||||||
|
)
|
||||||
|
payload_id = first_row.payload_id
|
||||||
|
disposed, evidence_hashes = self.provider.apply_retention(
|
||||||
|
self.session,
|
||||||
|
principal("admin", scopes=("datasources:source:admin",)),
|
||||||
|
as_of=as_of,
|
||||||
|
plan_hash=plan.plan_hash,
|
||||||
|
target_refs=(first.ref,),
|
||||||
|
)
|
||||||
|
self.assertEqual((first.ref,), disposed)
|
||||||
|
self.assertEqual(1, len(evidence_hashes))
|
||||||
|
self.assertIsNone(self.session.get(DatasourcePayloadRecord, payload_id))
|
||||||
|
self.assertIsNotNone(first_row.disposed_at)
|
||||||
|
self.assertEqual("disposed", first_row.state)
|
||||||
|
with self.assertRaises(DatasourceUnavailableError):
|
||||||
|
self.provider.read_datasource(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
request=DatasourceReadRequest(
|
||||||
|
datasource_ref=datasource.ref,
|
||||||
|
materialization_ref=first.ref,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
held = self.provider.update_datasource_governance(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
datasource_ref=datasource.ref,
|
||||||
|
governance=DatasourceGovernance(
|
||||||
|
retention_policy=governance.retention_policy,
|
||||||
|
hold_refs=("hold:legal-1",),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
frozen = self.provider.freeze_datasource(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
datasource_ref=held.ref,
|
||||||
|
label="Legal evidence",
|
||||||
|
)
|
||||||
|
held_plan = self.provider.preview_retention(
|
||||||
|
self.session,
|
||||||
|
principal("admin", scopes=("datasources:source:admin",)),
|
||||||
|
as_of=frozen.created_at + timedelta(days=2),
|
||||||
|
)
|
||||||
|
held_candidate = next(item for item in held_plan.candidates if item.ref == frozen.ref)
|
||||||
|
self.assertFalse(held_candidate.eligible)
|
||||||
|
self.assertIn("legal_hold", held_candidate.blockers)
|
||||||
|
|
||||||
|
def test_retention_deletes_only_an_explicitly_selected_eligible_stage(self) -> None:
|
||||||
|
stage = self.provider.create_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage=DatasourceStageInput(
|
||||||
|
name="Transient import",
|
||||||
|
source_name="transient_import",
|
||||||
|
kind="upload",
|
||||||
|
mode="static",
|
||||||
|
shape="tabular",
|
||||||
|
rows=({"id": 1},),
|
||||||
|
governance=DatasourceGovernance(
|
||||||
|
retention_policy={
|
||||||
|
"version": "stage-retention-v1",
|
||||||
|
"enabled": True,
|
||||||
|
"stage_days": 1,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
admin = principal(
|
||||||
|
"admin",
|
||||||
|
scopes=("datasources:source:admin",),
|
||||||
|
)
|
||||||
|
as_of = stage.created_at + timedelta(days=2)
|
||||||
|
plan = self.provider.preview_retention(
|
||||||
|
self.session,
|
||||||
|
admin,
|
||||||
|
as_of=as_of,
|
||||||
|
)
|
||||||
|
candidate = next(item for item in plan.candidates if item.ref == stage.ref)
|
||||||
|
self.assertTrue(candidate.eligible)
|
||||||
|
self.assertEqual("delete_stage", candidate.disposition)
|
||||||
|
|
||||||
|
disposed, evidence_hashes = self.provider.apply_retention(
|
||||||
|
self.session,
|
||||||
|
admin,
|
||||||
|
as_of=as_of,
|
||||||
|
plan_hash=plan.plan_hash,
|
||||||
|
target_refs=(stage.ref,),
|
||||||
|
)
|
||||||
|
self.assertEqual((stage.ref,), disposed)
|
||||||
|
self.assertEqual(1, len(evidence_hashes))
|
||||||
|
self.assertIsNone(
|
||||||
|
self.session.get(
|
||||||
|
DatasourceStageRecord,
|
||||||
|
stage.ref.removeprefix("stage:"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
evidence = self.provider.list_lifecycle_evidence(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
subject_ref=stage.ref,
|
||||||
|
)
|
||||||
|
self.assertEqual("retention.stage_deleted", evidence[0].event_type)
|
||||||
|
self.assertEqual(plan.plan_hash, evidence[0].details_["plan_hash"])
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
PAGE = ROOT / "webui" / "src" / "features" / "datasources" / "DatasourcesPage.tsx"
|
||||||
|
API = ROOT / "webui" / "src" / "api" / "datasources.ts"
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceLifecycleGovernanceUiTests(unittest.TestCase):
|
||||||
|
def test_stage_decisions_and_retention_use_shared_semantic_primitives(self) -> None:
|
||||||
|
source = PAGE.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
self.assertIn("<WorkspaceActionBar", source)
|
||||||
|
self.assertIn("function StageDecisionDialog", source)
|
||||||
|
self.assertIn("function RetentionDialog", source)
|
||||||
|
self.assertIn("<ConfirmDialog", source)
|
||||||
|
self.assertIn('variant="danger"', source)
|
||||||
|
self.assertIn("datasources.field.approval-policy", source)
|
||||||
|
self.assertIn("datasources.field.retention-policy-contract", source)
|
||||||
|
self.assertIn('state === "awaiting_approval"', source)
|
||||||
|
|
||||||
|
def test_client_binds_decisions_and_apply_to_server_evidence(self) -> None:
|
||||||
|
source = API.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
self.assertIn("expected_policy_hash", source)
|
||||||
|
self.assertIn("expected_subject_digest", source)
|
||||||
|
self.assertIn("/lifecycle-evidence", source)
|
||||||
|
self.assertIn("/retention/plan", source)
|
||||||
|
self.assertIn("plan_hash: plan.plan_hash", source)
|
||||||
|
self.assertIn("/refresh/stage", source)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -31,6 +31,10 @@ class DatasourceManifestTests(unittest.TestCase):
|
|||||||
if item.name == "connectors.datasource_origins"
|
if item.name == "connectors.datasource_origins"
|
||||||
)
|
)
|
||||||
self.assertTrue(requirement.optional)
|
self.assertTrue(requirement.optional)
|
||||||
|
self.assertIn(
|
||||||
|
"datasources:stage:approve",
|
||||||
|
{item.scope for item in manifest.permissions},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -34,7 +34,7 @@ class DatasourceMigrationTests(unittest.TestCase):
|
|||||||
try:
|
try:
|
||||||
with engine.connect() as connection:
|
with engine.connect() as connection:
|
||||||
self.assertIn(
|
self.assertIn(
|
||||||
"c9e3a6f1d4b8",
|
"d1a7c3e9f5b2",
|
||||||
set(MigrationContext.configure(connection).get_current_heads()),
|
set(MigrationContext.configure(connection).get_current_heads()),
|
||||||
)
|
)
|
||||||
catalogue_columns = {
|
catalogue_columns = {
|
||||||
@@ -52,13 +52,34 @@ class DatasourceMigrationTests(unittest.TestCase):
|
|||||||
"quality_policy",
|
"quality_policy",
|
||||||
"access_policy_ref",
|
"access_policy_ref",
|
||||||
"visibility_policy",
|
"visibility_policy",
|
||||||
|
"approval_policy",
|
||||||
|
"retention_policy",
|
||||||
"dependency_refs",
|
"dependency_refs",
|
||||||
}.issubset(catalogue_columns)
|
}.issubset(catalogue_columns)
|
||||||
)
|
)
|
||||||
|
stage_columns = {
|
||||||
|
item["name"]
|
||||||
|
for item in inspect(connection).get_columns(
|
||||||
|
"datasource_stages"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
materialization_columns = {
|
||||||
|
item["name"]
|
||||||
|
for item in inspect(connection).get_columns(
|
||||||
|
"datasource_materializations"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
self.assertIn("approval", stage_columns)
|
||||||
|
self.assertTrue(
|
||||||
|
{"disposed_at", "disposition"}.issubset(
|
||||||
|
materialization_columns
|
||||||
|
)
|
||||||
|
)
|
||||||
self.assertEqual(
|
self.assertEqual(
|
||||||
{
|
{
|
||||||
"datasource_catalogue",
|
"datasource_catalogue",
|
||||||
"datasource_governance_references",
|
"datasource_governance_references",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
"datasource_materializations",
|
"datasource_materializations",
|
||||||
"datasource_payload_rows",
|
"datasource_payload_rows",
|
||||||
"datasource_payloads",
|
"datasource_payloads",
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@govoplan/datasources-webui",
|
"name": "@govoplan/datasources-webui",
|
||||||
"version": "0.1.20",
|
"version": "0.1.21",
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
|
|||||||
@@ -35,6 +35,8 @@ export type DatasourceGovernance = {
|
|||||||
transfer_agreement_ref?: string | null;
|
transfer_agreement_ref?: string | null;
|
||||||
freshness_policy: Record<string, unknown>;
|
freshness_policy: Record<string, unknown>;
|
||||||
quality_policy: Record<string, unknown>;
|
quality_policy: Record<string, unknown>;
|
||||||
|
approval_policy: Record<string, unknown>;
|
||||||
|
retention_policy: Record<string, unknown>;
|
||||||
known_limits: string[];
|
known_limits: string[];
|
||||||
correction_procedure_ref?: string | null;
|
correction_procedure_ref?: string | null;
|
||||||
affected_refs: string[];
|
affected_refs: string[];
|
||||||
@@ -85,6 +87,8 @@ export type DatasourceMaterialization = {
|
|||||||
frozen_label?: string | null;
|
frozen_label?: string | null;
|
||||||
source_timestamp?: string | null;
|
source_timestamp?: string | null;
|
||||||
created_at?: string | null;
|
created_at?: string | null;
|
||||||
|
disposed_at?: string | null;
|
||||||
|
disposition: Record<string, unknown>;
|
||||||
provenance: Record<string, unknown>;
|
provenance: Record<string, unknown>;
|
||||||
metadata: Record<string, unknown>;
|
metadata: Record<string, unknown>;
|
||||||
governance: DatasourceGovernance;
|
governance: DatasourceGovernance;
|
||||||
@@ -142,6 +146,21 @@ export type DatasourceStage = {
|
|||||||
row_count?: number | null;
|
row_count?: number | null;
|
||||||
byte_count?: number | null;
|
byte_count?: number | null;
|
||||||
validation: DatasourceStageValidation;
|
validation: DatasourceStageValidation;
|
||||||
|
approval: {
|
||||||
|
state?: "not_required" | "pending" | "approved" | "rejected" | "expired";
|
||||||
|
policy_hash?: string;
|
||||||
|
subject_digest?: string;
|
||||||
|
required_approvals?: number;
|
||||||
|
approval_count?: number;
|
||||||
|
expires_at?: string | null;
|
||||||
|
approvals?: Array<{
|
||||||
|
actor_ref?: string;
|
||||||
|
decision?: "approve" | "reject";
|
||||||
|
reason?: string;
|
||||||
|
decided_at?: string;
|
||||||
|
}>;
|
||||||
|
policy?: Record<string, unknown>;
|
||||||
|
};
|
||||||
created_at?: string | null;
|
created_at?: string | null;
|
||||||
promoted_at?: string | null;
|
promoted_at?: string | null;
|
||||||
promoted_materialization_ref?: string | null;
|
promoted_materialization_ref?: string | null;
|
||||||
@@ -203,6 +222,38 @@ export type DatasourcePreview = {
|
|||||||
}>;
|
}>;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
export type DatasourceLifecycleEvidence = {
|
||||||
|
ref: string;
|
||||||
|
subject_ref: string;
|
||||||
|
event_type: string;
|
||||||
|
occurred_at: string;
|
||||||
|
actor_ref?: string | null;
|
||||||
|
policy_version?: string | null;
|
||||||
|
policy_hash?: string | null;
|
||||||
|
subject_digest: string;
|
||||||
|
previous_event_hash?: string | null;
|
||||||
|
event_hash: string;
|
||||||
|
details: Record<string, unknown>;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type DatasourceRetentionCandidate = {
|
||||||
|
ref: string;
|
||||||
|
kind: "stage" | "materialization";
|
||||||
|
datasource_ref?: string | null;
|
||||||
|
disposition: "delete_stage" | "purge_materialization_payload";
|
||||||
|
eligible_at: string;
|
||||||
|
eligible: boolean;
|
||||||
|
blockers: string[];
|
||||||
|
policy_version: string;
|
||||||
|
policy_hash: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type DatasourceRetentionPlan = {
|
||||||
|
as_of: string;
|
||||||
|
plan_hash: string;
|
||||||
|
candidates: DatasourceRetentionCandidate[];
|
||||||
|
};
|
||||||
|
|
||||||
export async function listDatasources(
|
export async function listDatasources(
|
||||||
settings: ApiSettings,
|
settings: ApiSettings,
|
||||||
query = "",
|
query = "",
|
||||||
@@ -293,6 +344,22 @@ export function promoteDatasourceStage(
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function decideDatasourceStage(
|
||||||
|
settings: ApiSettings,
|
||||||
|
stageRef: string,
|
||||||
|
payload: {
|
||||||
|
decision: "approve" | "reject";
|
||||||
|
reason: string;
|
||||||
|
expected_policy_hash: string;
|
||||||
|
expected_subject_digest: string;
|
||||||
|
}
|
||||||
|
): Promise<DatasourceStage> {
|
||||||
|
return apiFetch(settings, `/api/v1/datasources/stages/${refId(stageRef)}/decision`, {
|
||||||
|
method: "POST",
|
||||||
|
body: JSON.stringify(payload)
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
export function registerDatasourceOrigin(
|
export function registerDatasourceOrigin(
|
||||||
settings: ApiSettings,
|
settings: ApiSettings,
|
||||||
payload: {
|
payload: {
|
||||||
@@ -319,6 +386,50 @@ export function refreshDatasource(
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function prepareDatasourceRefresh(
|
||||||
|
settings: ApiSettings,
|
||||||
|
datasourceRef: string
|
||||||
|
): Promise<DatasourceStage> {
|
||||||
|
return apiFetch(settings, `/api/v1/datasources/${refId(datasourceRef)}/refresh/stage`, {
|
||||||
|
method: "POST"
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function listDatasourceLifecycleEvidence(
|
||||||
|
settings: ApiSettings,
|
||||||
|
subjectRef?: string
|
||||||
|
): Promise<DatasourceLifecycleEvidence[]> {
|
||||||
|
const params = new URLSearchParams();
|
||||||
|
if (subjectRef) params.set("subject_ref", subjectRef);
|
||||||
|
const suffix = params.size ? `?${params.toString()}` : "";
|
||||||
|
const response = await apiFetch<{ evidence: DatasourceLifecycleEvidence[] }>(
|
||||||
|
settings,
|
||||||
|
`/api/v1/datasources/lifecycle-evidence${suffix}`
|
||||||
|
);
|
||||||
|
return response.evidence;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function previewDatasourceRetention(
|
||||||
|
settings: ApiSettings
|
||||||
|
): Promise<DatasourceRetentionPlan> {
|
||||||
|
return apiFetch(settings, "/api/v1/datasources/retention/plan");
|
||||||
|
}
|
||||||
|
|
||||||
|
export function applyDatasourceRetention(
|
||||||
|
settings: ApiSettings,
|
||||||
|
plan: DatasourceRetentionPlan,
|
||||||
|
targetRefs: string[]
|
||||||
|
): Promise<{ plan_hash: string; disposed_refs: string[]; evidence_hashes: string[] }> {
|
||||||
|
return apiFetch(settings, "/api/v1/datasources/retention/apply", {
|
||||||
|
method: "POST",
|
||||||
|
body: JSON.stringify({
|
||||||
|
as_of: plan.as_of,
|
||||||
|
plan_hash: plan.plan_hash,
|
||||||
|
target_refs: targetRefs
|
||||||
|
})
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
export function freezeDatasource(
|
export function freezeDatasource(
|
||||||
settings: ApiSettings,
|
settings: ApiSettings,
|
||||||
datasourceRef: string,
|
datasourceRef: string,
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ import {
|
|||||||
Plus,
|
Plus,
|
||||||
Snowflake,
|
Snowflake,
|
||||||
ShieldCheck,
|
ShieldCheck,
|
||||||
|
ShieldQuestion,
|
||||||
Trash2,
|
Trash2,
|
||||||
Upload
|
Upload
|
||||||
} from "lucide-react";
|
} from "lucide-react";
|
||||||
@@ -49,14 +50,18 @@ import { FormGrid, DialogSection, ActionToolbar,
|
|||||||
type AuthInfo
|
type AuthInfo
|
||||||
} from "@govoplan/core-webui";
|
} from "@govoplan/core-webui";
|
||||||
import {
|
import {
|
||||||
|
applyDatasourceRetention,
|
||||||
createDatasourceStage,
|
createDatasourceStage,
|
||||||
|
decideDatasourceStage,
|
||||||
freezeDatasource,
|
freezeDatasource,
|
||||||
listDatasourceMaterializations,
|
listDatasourceMaterializations,
|
||||||
listDatasourceOrigins,
|
listDatasourceOrigins,
|
||||||
listDatasourceStages,
|
listDatasourceStages,
|
||||||
listDatasources,
|
listDatasources,
|
||||||
previewDatasource,
|
previewDatasource,
|
||||||
|
previewDatasourceRetention,
|
||||||
promoteDatasourceStage,
|
promoteDatasourceStage,
|
||||||
|
prepareDatasourceRefresh,
|
||||||
refreshDatasource,
|
refreshDatasource,
|
||||||
registerDatasourceOrigin,
|
registerDatasourceOrigin,
|
||||||
retireDatasource,
|
retireDatasource,
|
||||||
@@ -66,6 +71,7 @@ import {
|
|||||||
type DatasourceMaterialization,
|
type DatasourceMaterialization,
|
||||||
type DatasourceOrigin,
|
type DatasourceOrigin,
|
||||||
type DatasourcePreview,
|
type DatasourcePreview,
|
||||||
|
type DatasourceRetentionPlan,
|
||||||
type DatasourceSchemaChange,
|
type DatasourceSchemaChange,
|
||||||
type DatasourceStage,
|
type DatasourceStage,
|
||||||
type DatasourceValidationDiagnostic
|
type DatasourceValidationDiagnostic
|
||||||
@@ -113,12 +119,17 @@ export default function DatasourcesPage({
|
|||||||
const [retireOpen, setRetireOpen] = useState(false);
|
const [retireOpen, setRetireOpen] = useState(false);
|
||||||
const [promoteOpen, setPromoteOpen] = useState(false);
|
const [promoteOpen, setPromoteOpen] = useState(false);
|
||||||
const [governanceOpen, setGovernanceOpen] = useState(false);
|
const [governanceOpen, setGovernanceOpen] = useState(false);
|
||||||
|
const [decisionOpen, setDecisionOpen] = useState(false);
|
||||||
|
const [retentionOpen, setRetentionOpen] = useState(false);
|
||||||
const { requestDiscard } = useUnsavedChanges();
|
const { requestDiscard } = useUnsavedChanges();
|
||||||
|
|
||||||
const canManage = hasScope(auth, "datasources:source:write")
|
const canManage = hasScope(auth, "datasources:source:write")
|
||||||
|| hasScope(auth, "datasources:source:admin");
|
|| hasScope(auth, "datasources:source:admin");
|
||||||
const canStage = hasScope(auth, "datasources:stage:write")
|
const canStage = hasScope(auth, "datasources:stage:write")
|
||||||
|| hasScope(auth, "datasources:source:admin");
|
|| hasScope(auth, "datasources:source:admin");
|
||||||
|
const canApprove = hasScope(auth, "datasources:stage:approve")
|
||||||
|
|| hasScope(auth, "datasources:source:admin");
|
||||||
|
const canAdmin = hasScope(auth, "datasources:source:admin");
|
||||||
|
|
||||||
const reload = useCallback(async (preferredDatasourceRef?: string) => {
|
const reload = useCallback(async (preferredDatasourceRef?: string) => {
|
||||||
setLoading(true);
|
setLoading(true);
|
||||||
@@ -228,6 +239,14 @@ export default function DatasourcesPage({
|
|||||||
setWorking(true);
|
setWorking(true);
|
||||||
setError("");
|
setError("");
|
||||||
try {
|
try {
|
||||||
|
if (selectedDatasource.governance.approval_policy.required === true) {
|
||||||
|
const stage = await prepareDatasourceRefresh(settings, selectedDatasource.ref);
|
||||||
|
await reload(selectedDatasource.ref);
|
||||||
|
setSelectedStageRef(stage.ref);
|
||||||
|
setView("staging");
|
||||||
|
setSuccess(`Prepared refresh stage ${stage.name}. It must be approved before promotion.`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
const result = await refreshDatasource(settings, selectedDatasource.ref);
|
const result = await refreshDatasource(settings, selectedDatasource.ref);
|
||||||
setSuccess(`Refreshed ${result.datasource.name} as revision ${result.materialization.revision}.`);
|
setSuccess(`Refreshed ${result.datasource.name} as revision ${result.materialization.revision}.`);
|
||||||
await reload(result.datasource.ref);
|
await reload(result.datasource.ref);
|
||||||
@@ -425,7 +444,12 @@ export default function DatasourcesPage({
|
|||||||
primaryActions={<>
|
primaryActions={<>
|
||||||
{view === "catalogue" && selectedDatasource?.mode === "cached" ? (
|
{view === "catalogue" && selectedDatasource?.mode === "cached" ? (
|
||||||
<Button onClick={() => void refreshSelected()} disabled={!canManage || working} disabledReason={working ? DATASOURCES_I18N.working : !canManage ? DATASOURCES_I18N.manageReason : undefined}>
|
<Button onClick={() => void refreshSelected()} disabled={!canManage || working} disabledReason={working ? DATASOURCES_I18N.working : !canManage ? DATASOURCES_I18N.manageReason : undefined}>
|
||||||
<Download size={16} /> Refresh
|
<Download size={16} /> {selectedDatasource.governance.approval_policy.required === true ? "Stage refresh" : "Refresh"}
|
||||||
|
</Button>
|
||||||
|
) : null}
|
||||||
|
{view === "catalogue" && canAdmin ? (
|
||||||
|
<Button onClick={() => setRetentionOpen(true)} disabled={working}>
|
||||||
|
<Archive size={16} /> Retention
|
||||||
</Button>
|
</Button>
|
||||||
) : null}
|
) : null}
|
||||||
{view === "catalogue" && selectedDatasource ? (
|
{view === "catalogue" && selectedDatasource ? (
|
||||||
@@ -452,6 +476,16 @@ export default function DatasourcesPage({
|
|||||||
<Upload size={16} /> Promote
|
<Upload size={16} /> Promote
|
||||||
</Button>
|
</Button>
|
||||||
) : null}
|
) : null}
|
||||||
|
{view === "staging" && selectedStage?.state === "awaiting_approval" ? (
|
||||||
|
<Button
|
||||||
|
variant="primary"
|
||||||
|
onClick={() => setDecisionOpen(true)}
|
||||||
|
disabled={!canApprove || working}
|
||||||
|
disabledReason={working ? DATASOURCES_I18N.working : !canApprove ? "Datasource approval permission is required." : undefined}
|
||||||
|
>
|
||||||
|
<ShieldQuestion size={16} /> Decide
|
||||||
|
</Button>
|
||||||
|
) : null}
|
||||||
{view === "origins" && selectedOrigin ? (
|
{view === "origins" && selectedOrigin ? (
|
||||||
<Button
|
<Button
|
||||||
variant="primary"
|
variant="primary"
|
||||||
@@ -563,6 +597,28 @@ export default function DatasourcesPage({
|
|||||||
await reload(updated.ref);
|
await reload(updated.ref);
|
||||||
}}
|
}}
|
||||||
/>
|
/>
|
||||||
|
<StageDecisionDialog
|
||||||
|
open={decisionOpen}
|
||||||
|
settings={settings}
|
||||||
|
stage={selectedStage}
|
||||||
|
onClose={() => setDecisionOpen(false)}
|
||||||
|
onDecided={async (updated) => {
|
||||||
|
setDecisionOpen(false);
|
||||||
|
await reload();
|
||||||
|
setSelectedStageRef(updated.ref);
|
||||||
|
setSuccess(`Recorded ${updated.approval.state ?? "approval"} decision state for ${updated.name}.`);
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
<RetentionDialog
|
||||||
|
open={retentionOpen}
|
||||||
|
settings={settings}
|
||||||
|
onClose={() => setRetentionOpen(false)}
|
||||||
|
onApplied={async (count) => {
|
||||||
|
setRetentionOpen(false);
|
||||||
|
setSuccess(`Applied retention to ${count} eligible target${count === 1 ? "" : "s"}.`);
|
||||||
|
await reload(selectedDatasourceRef);
|
||||||
|
}}
|
||||||
|
/>
|
||||||
<Dialog
|
<Dialog
|
||||||
open={freezeOpen}
|
open={freezeOpen}
|
||||||
title="Freeze datasource state"
|
title="Freeze datasource state"
|
||||||
@@ -651,6 +707,8 @@ function DatasourceDetail({
|
|||||||
<span><small>Steward</small><strong>{datasource.governance.steward_ref || "Not assigned"}</strong></span>
|
<span><small>Steward</small><strong>{datasource.governance.steward_ref || "Not assigned"}</strong></span>
|
||||||
<span><small>Responsible organization</small><strong>{datasource.governance.responsible_organization_ref || "Not assigned"}</strong></span>
|
<span><small>Responsible organization</small><strong>{datasource.governance.responsible_organization_ref || "Not assigned"}</strong></span>
|
||||||
<span><small>Purposes</small><strong>{datasource.governance.purposes.join(", ") || "Not declared"}</strong></span>
|
<span><small>Purposes</small><strong>{datasource.governance.purposes.join(", ") || "Not declared"}</strong></span>
|
||||||
|
<span><small>Approval gate</small><strong>{datasource.governance.approval_policy.required === true ? "Required" : "Not required"}</strong></span>
|
||||||
|
<span><small>Retention</small><strong>{datasource.governance.retention_policy.enabled === true ? "Configured" : "Disabled"}</strong></span>
|
||||||
</div>
|
</div>
|
||||||
{datasource.governance.semantic_definition ? (
|
{datasource.governance.semantic_definition ? (
|
||||||
<p className="datasources-dialog-copy">{datasource.governance.semantic_definition}</p>
|
<p className="datasources-dialog-copy">{datasource.governance.semantic_definition}</p>
|
||||||
@@ -692,7 +750,9 @@ function DatasourceDetail({
|
|||||||
<td>{formatDate(item.created_at)}</td>
|
<td>{formatDate(item.created_at)}</td>
|
||||||
<td>{formatNumber(item.row_count)}</td>
|
<td>{formatNumber(item.row_count)}</td>
|
||||||
<td>
|
<td>
|
||||||
{item.frozen_at ? (
|
{item.disposed_at ? (
|
||||||
|
<StatusBadge status="disposed" label="payload disposed" />
|
||||||
|
) : item.frozen_at ? (
|
||||||
<StatusBadge
|
<StatusBadge
|
||||||
status="frozen"
|
status="frozen"
|
||||||
label={item.frozen_label || "frozen"}
|
label={item.frozen_label || "frozen"}
|
||||||
@@ -773,6 +833,28 @@ function StageDetail({ stage }: { stage: DatasourceStage }) {
|
|||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
</ContentSection>
|
</ContentSection>
|
||||||
|
<ContentSection>
|
||||||
|
<ActionToolbar surface="section-header" className="datasources-section-heading">
|
||||||
|
<span><ShieldQuestion size={16} /> Promotion approval</span>
|
||||||
|
<StatusBadge status={stage.approval.state ?? "not_required"} label={readableToken(stage.approval.state ?? "not_required")} />
|
||||||
|
</ActionToolbar>
|
||||||
|
<div className="datasources-key-values">
|
||||||
|
<span><small>Progress</small><strong>{stage.approval.approval_count ?? 0} / {stage.approval.required_approvals ?? 1}</strong></span>
|
||||||
|
<span><small>Policy hash</small><strong>{shortFingerprint(stage.approval.policy_hash ?? "")}</strong></span>
|
||||||
|
<span><small>Subject digest</small><strong>{shortFingerprint(stage.approval.subject_digest ?? "")}</strong></span>
|
||||||
|
<span><small>Expires</small><strong>{formatDate(stage.approval.expires_at)}</strong></span>
|
||||||
|
</div>
|
||||||
|
{stage.approval.approvals?.length ? (
|
||||||
|
<ul className="datasources-diagnostic-list">
|
||||||
|
{stage.approval.approvals.map((item, index) => (
|
||||||
|
<li key={`${item.actor_ref ?? "actor"}-${index}`}>
|
||||||
|
<span>{item.actor_ref ?? "Unknown actor"} · {readableToken(item.decision ?? "decision")}</span>
|
||||||
|
<small>{item.reason ?? "No reason recorded"} · {formatDate(item.decided_at)}</small>
|
||||||
|
</li>
|
||||||
|
))}
|
||||||
|
</ul>
|
||||||
|
) : <div className="datasources-validation-ok">No approval decision has been recorded.</div>}
|
||||||
|
</ContentSection>
|
||||||
{schemaChanges.length ? (
|
{schemaChanges.length ? (
|
||||||
<ContentSection>
|
<ContentSection>
|
||||||
<ActionToolbar surface="section-header" className="datasources-section-heading">
|
<ActionToolbar surface="section-header" className="datasources-section-heading">
|
||||||
@@ -797,6 +879,211 @@ function StageDetail({ stage }: { stage: DatasourceStage }) {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function StageDecisionDialog({
|
||||||
|
open,
|
||||||
|
settings,
|
||||||
|
stage,
|
||||||
|
onClose,
|
||||||
|
onDecided
|
||||||
|
}: {
|
||||||
|
open: boolean;
|
||||||
|
settings: ApiSettings;
|
||||||
|
stage: DatasourceStage | null;
|
||||||
|
onClose: () => void;
|
||||||
|
onDecided: (stage: DatasourceStage) => void | Promise<void>;
|
||||||
|
}) {
|
||||||
|
const [decision, setDecision] = useState<"approve" | "reject">("approve");
|
||||||
|
const [reason, setReason] = useState("");
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
const [error, setError] = useState("");
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!open) return;
|
||||||
|
setDecision("approve");
|
||||||
|
setReason("");
|
||||||
|
setError("");
|
||||||
|
}, [open, stage?.ref]);
|
||||||
|
|
||||||
|
const submit = async () => {
|
||||||
|
if (!stage?.approval.policy_hash || !stage.approval.subject_digest) return;
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const updated = await decideDatasourceStage(settings, stage.ref, {
|
||||||
|
decision,
|
||||||
|
reason,
|
||||||
|
expected_policy_hash: stage.approval.policy_hash,
|
||||||
|
expected_subject_digest: stage.approval.subject_digest
|
||||||
|
});
|
||||||
|
await onDecided(updated);
|
||||||
|
} catch (operationError) {
|
||||||
|
setError(apiErrorMessage(operationError));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
return (
|
||||||
|
<Dialog
|
||||||
|
open={open}
|
||||||
|
title="Decide datasource promotion"
|
||||||
|
onClose={() => !busy && onClose()}
|
||||||
|
footer={<>
|
||||||
|
<Button onClick={onClose} disabled={busy}>Cancel</Button>
|
||||||
|
<Button
|
||||||
|
variant={decision === "reject" ? "danger" : "primary"}
|
||||||
|
onClick={() => void submit()}
|
||||||
|
disabled={busy || reason.trim().length < 5 || !stage?.approval.policy_hash}
|
||||||
|
>
|
||||||
|
{decision === "approve" ? "Approve stage" : "Reject stage"}
|
||||||
|
</Button>
|
||||||
|
</>}
|
||||||
|
>
|
||||||
|
{error ? <DismissibleAlert tone="danger" resetKey={error}>{error}</DismissibleAlert> : null}
|
||||||
|
<p className="datasources-dialog-copy">
|
||||||
|
The decision is bound to this exact stage, validation result, policy hash, and your account authority.
|
||||||
|
</p>
|
||||||
|
<div className="datasources-key-values">
|
||||||
|
<span><small>Stage</small><strong>{stage?.name ?? "Unavailable"}</strong></span>
|
||||||
|
<span><small>Approval progress</small><strong>{stage?.approval.approval_count ?? 0} / {stage?.approval.required_approvals ?? 1}</strong></span>
|
||||||
|
<span><small>Policy hash</small><strong>{shortFingerprint(stage?.approval.policy_hash ?? "")}</strong></span>
|
||||||
|
<span><small>Expires</small><strong>{formatDate(stage?.approval.expires_at)}</strong></span>
|
||||||
|
</div>
|
||||||
|
<SegmentedControl<"approve" | "reject">
|
||||||
|
ariaLabel="Approval decision"
|
||||||
|
width="fill"
|
||||||
|
size="equal"
|
||||||
|
options={[
|
||||||
|
{ id: "approve", label: "Approve" },
|
||||||
|
{ id: "reject", label: "Reject" }
|
||||||
|
]}
|
||||||
|
value={decision}
|
||||||
|
onChange={setDecision}
|
||||||
|
/>
|
||||||
|
<FormField label="Decision reason" interfaceId="datasources.action.approve" helpContextId="datasources.action.approve" helpModuleId="datasources">
|
||||||
|
<textarea value={reason} onChange={(event) => setReason(event.target.value)} placeholder="Record what was reviewed and why this decision is justified." />
|
||||||
|
</FormField>
|
||||||
|
</Dialog>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function RetentionDialog({
|
||||||
|
open,
|
||||||
|
settings,
|
||||||
|
onClose,
|
||||||
|
onApplied
|
||||||
|
}: {
|
||||||
|
open: boolean;
|
||||||
|
settings: ApiSettings;
|
||||||
|
onClose: () => void;
|
||||||
|
onApplied: (count: number) => void | Promise<void>;
|
||||||
|
}) {
|
||||||
|
const [plan, setPlan] = useState<DatasourceRetentionPlan | null>(null);
|
||||||
|
const [selected, setSelected] = useState<string[]>([]);
|
||||||
|
const [loading, setLoading] = useState(false);
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
const [confirming, setConfirming] = useState(false);
|
||||||
|
const [error, setError] = useState("");
|
||||||
|
|
||||||
|
const loadPlan = useCallback(async () => {
|
||||||
|
setLoading(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const next = await previewDatasourceRetention(settings);
|
||||||
|
setPlan(next);
|
||||||
|
setSelected([]);
|
||||||
|
} catch (loadError) {
|
||||||
|
setError(apiErrorMessage(loadError));
|
||||||
|
} finally {
|
||||||
|
setLoading(false);
|
||||||
|
}
|
||||||
|
}, [settings]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (open) void loadPlan();
|
||||||
|
}, [loadPlan, open]);
|
||||||
|
|
||||||
|
const apply = async () => {
|
||||||
|
if (!plan || !selected.length) return;
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const result = await applyDatasourceRetention(settings, plan, selected);
|
||||||
|
setConfirming(false);
|
||||||
|
await onApplied(result.disposed_refs.length);
|
||||||
|
} catch (operationError) {
|
||||||
|
setConfirming(false);
|
||||||
|
setError(apiErrorMessage(operationError));
|
||||||
|
await loadPlan();
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const toggle = (ref: string) => setSelected((current) =>
|
||||||
|
current.includes(ref)
|
||||||
|
? current.filter((item) => item !== ref)
|
||||||
|
: [...current, ref]);
|
||||||
|
|
||||||
|
return <>
|
||||||
|
<Dialog
|
||||||
|
open={open}
|
||||||
|
title="Datasource retention preview"
|
||||||
|
size="wide"
|
||||||
|
onClose={() => !busy && onClose()}
|
||||||
|
footer={<>
|
||||||
|
<Button onClick={() => void loadPlan()} disabled={loading || busy}>Reload preview</Button>
|
||||||
|
<Button onClick={onClose} disabled={busy}>Cancel</Button>
|
||||||
|
<Button variant="danger" onClick={() => setConfirming(true)} disabled={busy || loading || !selected.length}>
|
||||||
|
Apply selected dispositions
|
||||||
|
</Button>
|
||||||
|
</>}
|
||||||
|
>
|
||||||
|
{error ? <DismissibleAlert tone="danger" resetKey={error}>{error}</DismissibleAlert> : null}
|
||||||
|
<p className="datasources-dialog-copy">
|
||||||
|
The preview is read-only. Legal holds, current materializations, pending approvals, and publication evidence remain blocked.
|
||||||
|
</p>
|
||||||
|
{plan ? (
|
||||||
|
<div className="datasources-key-values">
|
||||||
|
<span><small>As of</small><strong>{formatDate(plan.as_of)}</strong></span>
|
||||||
|
<span><small>Plan hash</small><strong>{shortFingerprint(plan.plan_hash)}</strong></span>
|
||||||
|
<span><small>Due targets</small><strong>{plan.candidates.length}</strong></span>
|
||||||
|
<span><small>Selected</small><strong>{selected.length}</strong></span>
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
<LoadingFrame loading={loading}>
|
||||||
|
<div className="datasources-table-scroll">
|
||||||
|
<table>
|
||||||
|
<thead><tr><th className="datasources-retention-select-cell" aria-label="Select" /><th>Target</th><th>Disposition</th><th>Eligible since</th><th>Policy / blockers</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
{plan?.candidates.map((candidate) => (
|
||||||
|
<tr key={candidate.ref}>
|
||||||
|
<td className="datasources-retention-select-cell"><input type="checkbox" aria-label={`Select ${candidate.ref}`} checked={selected.includes(candidate.ref)} disabled={!candidate.eligible} onChange={() => toggle(candidate.ref)} /></td>
|
||||||
|
<td><strong>{candidate.ref}</strong><small>{candidate.datasource_ref ?? "Unpromoted stage"}</small></td>
|
||||||
|
<td>{readableToken(candidate.disposition)}</td>
|
||||||
|
<td>{formatDate(candidate.eligible_at)}</td>
|
||||||
|
<td>{candidate.blockers.length ? candidate.blockers.map(readableToken).join(", ") : `v${candidate.policy_version} · eligible`}</td>
|
||||||
|
</tr>
|
||||||
|
))}
|
||||||
|
{!plan?.candidates.length ? <tr><td colSpan={5}>No retention targets are due.</td></tr> : null}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
</LoadingFrame>
|
||||||
|
</Dialog>
|
||||||
|
<ConfirmDialog
|
||||||
|
open={confirming}
|
||||||
|
title="Apply datasource retention"
|
||||||
|
message={`Dispose ${selected.length} selected target${selected.length === 1 ? "" : "s"}? Payload rows or transient stages will be removed; lifecycle evidence remains.`}
|
||||||
|
confirmLabel="Apply retention"
|
||||||
|
tone="danger"
|
||||||
|
busy={busy}
|
||||||
|
onCancel={() => setConfirming(false)}
|
||||||
|
onConfirm={() => void apply()}
|
||||||
|
/>
|
||||||
|
</>;
|
||||||
|
}
|
||||||
|
|
||||||
function ValidationDiagnosticList({
|
function ValidationDiagnosticList({
|
||||||
diagnostics
|
diagnostics
|
||||||
}: {
|
}: {
|
||||||
@@ -900,6 +1187,8 @@ function GovernanceDialog({
|
|||||||
const [freshness, setFreshness] = useState("{}");
|
const [freshness, setFreshness] = useState("{}");
|
||||||
const [quality, setQuality] = useState("{}");
|
const [quality, setQuality] = useState("{}");
|
||||||
const [visibility, setVisibility] = useState("{}");
|
const [visibility, setVisibility] = useState("{}");
|
||||||
|
const [approval, setApproval] = useState("{}");
|
||||||
|
const [retention, setRetention] = useState("{}");
|
||||||
const [baselineKey, setBaselineKey] = useState("");
|
const [baselineKey, setBaselineKey] = useState("");
|
||||||
const [busy, setBusy] = useState(false);
|
const [busy, setBusy] = useState(false);
|
||||||
const [error, setError] = useState("");
|
const [error, setError] = useState("");
|
||||||
@@ -911,15 +1200,19 @@ function GovernanceDialog({
|
|||||||
const nextFreshness = JSON.stringify(datasource.governance.freshness_policy, null, 2);
|
const nextFreshness = JSON.stringify(datasource.governance.freshness_policy, null, 2);
|
||||||
const nextQuality = JSON.stringify(datasource.governance.quality_policy, null, 2);
|
const nextQuality = JSON.stringify(datasource.governance.quality_policy, null, 2);
|
||||||
const nextVisibility = JSON.stringify(datasource.governance.visibility_policy ?? {}, null, 2);
|
const nextVisibility = JSON.stringify(datasource.governance.visibility_policy ?? {}, null, 2);
|
||||||
|
const nextApproval = JSON.stringify(datasource.governance.approval_policy ?? {}, null, 2);
|
||||||
|
const nextRetention = JSON.stringify(datasource.governance.retention_policy ?? {}, null, 2);
|
||||||
setDraft(nextDraft);
|
setDraft(nextDraft);
|
||||||
setFreshness(nextFreshness);
|
setFreshness(nextFreshness);
|
||||||
setQuality(nextQuality);
|
setQuality(nextQuality);
|
||||||
setVisibility(nextVisibility);
|
setVisibility(nextVisibility);
|
||||||
setBaselineKey(JSON.stringify({ draft: nextDraft, freshness: nextFreshness, quality: nextQuality, visibility: nextVisibility }));
|
setApproval(nextApproval);
|
||||||
|
setRetention(nextRetention);
|
||||||
|
setBaselineKey(JSON.stringify({ draft: nextDraft, freshness: nextFreshness, quality: nextQuality, visibility: nextVisibility, approval: nextApproval, retention: nextRetention }));
|
||||||
setError("");
|
setError("");
|
||||||
}, [datasource, open]);
|
}, [datasource, open]);
|
||||||
|
|
||||||
const dirty = Boolean(open && draft && JSON.stringify({ draft, freshness, quality, visibility }) !== baselineKey);
|
const dirty = Boolean(open && draft && JSON.stringify({ draft, freshness, quality, visibility, approval, retention }) !== baselineKey);
|
||||||
|
|
||||||
const save = async (): Promise<boolean> => {
|
const save = async (): Promise<boolean> => {
|
||||||
if (!datasource || !draft) return false;
|
if (!datasource || !draft) return false;
|
||||||
@@ -930,7 +1223,9 @@ function GovernanceDialog({
|
|||||||
...draft,
|
...draft,
|
||||||
freshness_policy: parseObject(freshness, "Freshness policy"),
|
freshness_policy: parseObject(freshness, "Freshness policy"),
|
||||||
quality_policy: parseObject(quality, "Quality policy"),
|
quality_policy: parseObject(quality, "Quality policy"),
|
||||||
visibility_policy: parseObject(visibility, "Visibility policy")
|
visibility_policy: parseObject(visibility, "Visibility policy"),
|
||||||
|
approval_policy: parseObject(approval, "Approval policy"),
|
||||||
|
retention_policy: parseObject(retention, "Retention policy")
|
||||||
});
|
});
|
||||||
await onSaved(updated);
|
await onSaved(updated);
|
||||||
return true;
|
return true;
|
||||||
@@ -1058,6 +1353,14 @@ function GovernanceDialog({
|
|||||||
<FormField label="Quality policy (JSON)" documentation={DATASOURCE_GOVERNANCE_DOCUMENTATION}>
|
<FormField label="Quality policy (JSON)" documentation={DATASOURCE_GOVERNANCE_DOCUMENTATION}>
|
||||||
<textarea value={quality} onChange={(event) => setQuality(event.target.value)} spellCheck={false} />
|
<textarea value={quality} onChange={(event) => setQuality(event.target.value)} spellCheck={false} />
|
||||||
</FormField>
|
</FormField>
|
||||||
|
<FormField label="Approval policy (JSON)" interfaceId="datasources.field.approval-policy" helpContextId="datasources.field.approval-policy" helpModuleId="datasources" documentation={DATASOURCE_GOVERNANCE_DOCUMENTATION}>
|
||||||
|
<textarea value={approval} onChange={(event) => setApproval(event.target.value)} spellCheck={false} />
|
||||||
|
<small>Configure a version, required approvals, separation of duties, and optional expiry.</small>
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Retention policy (JSON)" interfaceId="datasources.field.retention-policy-contract" helpContextId="datasources.field.retention-policy-contract" helpModuleId="datasources" documentation={DATASOURCE_GOVERNANCE_DOCUMENTATION}>
|
||||||
|
<textarea value={retention} onChange={(event) => setRetention(event.target.value)} spellCheck={false} />
|
||||||
|
<small>Configure explicit stage, materialization, and frozen-evidence durations; execution always starts with a preview.</small>
|
||||||
|
</FormField>
|
||||||
</FormGrid>
|
</FormGrid>
|
||||||
</DialogSection>
|
</DialogSection>
|
||||||
) : null}
|
) : null}
|
||||||
|
|||||||
@@ -287,6 +287,17 @@
|
|||||||
max-height: min(860px, calc(100vh - 32px));
|
max-height: min(860px, calc(100vh - 32px));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.datasources-retention-select-cell {
|
||||||
|
width: 2.5rem;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
.datasources-table-scroll td small {
|
||||||
|
display: block;
|
||||||
|
margin-top: 0.2rem;
|
||||||
|
color: var(--color-text-muted);
|
||||||
|
}
|
||||||
|
|
||||||
.datasources-governance-dialog .datasources-dialog-fields textarea {
|
.datasources-governance-dialog .datasources-dialog-fields textarea {
|
||||||
min-height: 76px;
|
min-height: 76px;
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user