Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1f338cae70 | ||
|
|
9d067f1bad | ||
|
|
8b8c6c548e | ||
|
|
930302b535 | ||
|
|
ad8235028f | ||
|
|
62970d6374 | ||
|
|
6b1c0ca786 | ||
|
|
7a7654cc0f | ||
|
|
b54d1919e4 |
@@ -38,6 +38,14 @@ promotion preserves the exact policy hash and validation result in immutable
|
|||||||
materialization provenance. The supported contract is documented in
|
materialization provenance. The supported contract is documented in
|
||||||
[docs/QUALITY_POLICY.md](docs/QUALITY_POLICY.md).
|
[docs/QUALITY_POLICY.md](docs/QUALITY_POLICY.md).
|
||||||
|
|
||||||
|
The same lifecycle contract can require an attributable, separated approval
|
||||||
|
quorum before a stage or cached refresh becomes current. Versioned retention
|
||||||
|
rules cover transient stages, ordinary materializations, and frozen evidence.
|
||||||
|
Retention remains an explicit preview-and-apply operation: current revisions,
|
||||||
|
legal holds, pending approvals, and publication evidence are blocked, while
|
||||||
|
every decision, promotion, and disposition is preserved in hash-chained local
|
||||||
|
evidence even when the optional Audit, Access, or Policy modules are absent.
|
||||||
|
|
||||||
The contracts already model database, HTTP/REST, directory, file, feed,
|
The contracts already model database, HTTP/REST, directory, file, feed,
|
||||||
document, binary, directory, and stream sources so providers can be added
|
document, binary, directory, and stream sources so providers can be added
|
||||||
without changing consumers. Storage modules contribute artifact backends
|
without changing consumers. Storage modules contribute artifact backends
|
||||||
@@ -49,7 +57,8 @@ See [docs/CONCEPT.md](docs/CONCEPT.md) for ownership and lifecycle details.
|
|||||||
## Data-subject requests
|
## Data-subject requests
|
||||||
|
|
||||||
Datasources publishes `privacy.dsar.datasources` for exact catalogue,
|
Datasources publishes `privacy.dsar.datasources` for exact catalogue,
|
||||||
governance-reference, materialization, payload, stage, and publication
|
governance-reference, materialization, payload, stage, publication, and
|
||||||
|
lifecycle-evidence
|
||||||
references and for minimized operator attribution. It never exports connector
|
references and for minimized operator attribution. It never exports connector
|
||||||
references, locators, credentials, arbitrary rows, schemas, validation
|
references, locators, credentials, arbitrary rows, schemas, validation
|
||||||
samples, metadata, provenance bodies, checkpoints, replay material, or hashes.
|
samples, metadata, provenance bodies, checkpoints, replay material, or hashes.
|
||||||
@@ -57,6 +66,25 @@ The module does not guess subject identity by scanning schema-dependent tabular
|
|||||||
payloads; the authoritative source module locates and corrects those facts.
|
payloads; the authoritative source module locates and corrects those facts.
|
||||||
|
|
||||||
Unpromoted stages and unreferenced payloads can be deleted idempotently.
|
Unpromoted stages and unreferenced payloads can be deleted idempotently.
|
||||||
Published or referenced state, immutable materializations, governance evidence,
|
Published or referenced state, immutable materializations, lifecycle evidence,
|
||||||
|
governance evidence,
|
||||||
holds, and operator attribution require data-steward review. Dataflow and
|
holds, and operator attribution require data-steward review. Dataflow and
|
||||||
Reporting derivatives must be refreshed after the source correction.
|
Reporting derivatives must be refreshed after the source correction.
|
||||||
|
|
||||||
|
## Git-source WebUI package
|
||||||
|
|
||||||
|
The repository root exposes `@govoplan/datasources-webui` for Git-tagged release
|
||||||
|
dependencies. It mirrors the owning `webui/package.json` version, public
|
||||||
|
TypeScript/CSS exports and peer requirements, with entry paths under
|
||||||
|
`webui/src`. Consumers provide the shared Core/React peers; the facade runs no
|
||||||
|
development or install scripts. The source archive contains `webui/src`, this
|
||||||
|
README and any repository license file. Run module development checks from `webui/`; Python
|
||||||
|
installation remains governed by `pyproject.toml`.
|
||||||
|
|
||||||
|
Das Repository stellt `@govoplan/datasources-webui` am Wurzelpfad für versionierte
|
||||||
|
Git-Abhängigkeiten bereit. Version, öffentliche TypeScript-/CSS-Exporte und
|
||||||
|
Peer-Anforderungen entsprechen `webui/package.json`; die Einstiegspfade liegen
|
||||||
|
unter `webui/src`. Gemeinsame Core-/React-Peers stellt die einbindende Anwendung
|
||||||
|
bereit. Die Fassade führt keine Entwicklungs- oder Installationsskripte aus.
|
||||||
|
Entwicklungsprüfungen bleiben in `webui/`, die Python-Installation weiterhin in
|
||||||
|
`pyproject.toml` definiert.
|
||||||
|
|||||||
@@ -88,6 +88,21 @@ The first slice stores bounded tabular JSON/CSV stages. Future providers may
|
|||||||
stage file references, object-store blobs, directory snapshots, or streaming
|
stage file references, object-store blobs, directory snapshots, or streaming
|
||||||
checkpoints through the same lifecycle contract.
|
checkpoints through the same lifecycle contract.
|
||||||
|
|
||||||
|
Valid stages may be subject to a versioned local approval policy. Required
|
||||||
|
quorums keep the stage non-consumable, enforce distinct attributable actors and
|
||||||
|
optional creator/approver separation, and bind every decision to exact policy
|
||||||
|
and subject hashes. Cached refreshes use this same staging path when approval is
|
||||||
|
required. An installation without separate Access or Policy modules retains
|
||||||
|
safe local scope and policy defaults; optional modules do not manufacture an
|
||||||
|
approval from configuration metadata.
|
||||||
|
|
||||||
|
Retention policy separately defines durations for stages, ordinary
|
||||||
|
materializations, and frozen evidence. Administrators receive a deterministic
|
||||||
|
plan before applying any disposition. Holds, current state, pending decisions,
|
||||||
|
and publication evidence are blockers. Payload disposal preserves a minimized
|
||||||
|
materialization tombstone and hash-chained lifecycle evidence, so chronological
|
||||||
|
authority and policy proof survive the configured content-retention boundary.
|
||||||
|
|
||||||
## Consumer Contract
|
## Consumer Contract
|
||||||
|
|
||||||
Consumers request:
|
Consumers request:
|
||||||
|
|||||||
+56
-3
@@ -116,6 +116,59 @@ replace the Datasource's current state. Valid warnings produce
|
|||||||
catalogue side effect. These terminal states are preserved for Dataflow and
|
catalogue side effect. These terminal states are preserved for Dataflow and
|
||||||
Workflow handoffs instead of being collapsed into generic success.
|
Workflow handoffs instead of being collapsed into generic success.
|
||||||
|
|
||||||
Approval authority, approval expiry, and retention/deletion execution remain
|
## Promotion approval
|
||||||
separate work under `govoplan-datasources#2`. Until those contracts are added,
|
|
||||||
no JSON flag is treated as an approval and no stage is deleted automatically.
|
Approval is a separate deterministic contract in
|
||||||
|
`governance.approval_policy`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"version": "monthly-promotion-v2",
|
||||||
|
"required": true,
|
||||||
|
"required_approvals": 2,
|
||||||
|
"separation_of_duties": true,
|
||||||
|
"expires_after_hours": 72,
|
||||||
|
"policy_ref": "policy:monthly-register-promotion"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
A valid stage enters `awaiting_approval` instead of `ready`. Each decision is
|
||||||
|
bound to the stage fingerprint, quality-policy hash, normalized approval-policy
|
||||||
|
hash, actor authority, reason, and expiry. Actors are distinct and the stage
|
||||||
|
creator cannot approve when separation of duties is enabled. The exact quorum
|
||||||
|
snapshot and hash-chained evidence are copied into materialization provenance.
|
||||||
|
A changed target policy or staged subject requires a new stage; request JSON
|
||||||
|
cannot claim that an approval occurred.
|
||||||
|
|
||||||
|
Cached origins use `POST /datasources/{id}/refresh/stage` when approval is
|
||||||
|
required. Direct refresh then fails closed, so connector content cannot become
|
||||||
|
current before the staged validation and approval quorum succeed.
|
||||||
|
|
||||||
|
## Retention
|
||||||
|
|
||||||
|
Retention is independently configured in `governance.retention_policy`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"version": "register-retention-v3",
|
||||||
|
"enabled": true,
|
||||||
|
"stage_days": 30,
|
||||||
|
"materialization_days": 365,
|
||||||
|
"frozen_evidence_days": 3650,
|
||||||
|
"policy_ref": "policy:register-retention"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Durations are optional; omitting a class retains it indefinitely. Empty or
|
||||||
|
disabled local policy never deletes content. An administrator first requests a
|
||||||
|
read-only plan. The plan includes all due targets, policy versions and hashes,
|
||||||
|
eligibility dates, blockers, and one hash over the complete preview. Applying
|
||||||
|
retention accepts only targets from the unchanged plan.
|
||||||
|
|
||||||
|
Pending approvals, current materializations, legal holds, and materializations
|
||||||
|
referenced by producer publications are blocked. Eligible stages are deleted.
|
||||||
|
Eligible materialization payload rows are purged while the revision retains its
|
||||||
|
schema, provenance, row/byte counts, payload checksum, disposition metadata,
|
||||||
|
and immutable hash-chained lifecycle evidence. Frozen evidence is eligible only
|
||||||
|
when `frozen_evidence_days` is explicitly configured. No hidden scheduler or
|
||||||
|
arbitrary Policy/Access flag performs deletion.
|
||||||
|
|||||||
@@ -0,0 +1,34 @@
|
|||||||
|
{
|
||||||
|
"name": "@govoplan/datasources-webui",
|
||||||
|
"version": "0.1.26",
|
||||||
|
"private": true,
|
||||||
|
"type": "module",
|
||||||
|
"main": "webui/src/index.ts",
|
||||||
|
"module": "webui/src/index.ts",
|
||||||
|
"types": "webui/src/index.ts",
|
||||||
|
"exports": {
|
||||||
|
".": {
|
||||||
|
"types": "./webui/src/index.ts",
|
||||||
|
"import": "./webui/src/index.ts"
|
||||||
|
},
|
||||||
|
"./styles/datasources.css": "./webui/src/styles/datasources.css"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"@govoplan/core-webui": "^0.1.46",
|
||||||
|
"lucide-react": "^1.23.0",
|
||||||
|
"react": ">=19.2.7 <20",
|
||||||
|
"react-dom": ">=19.2.7 <20",
|
||||||
|
"react-router": ">=8.3.0 <9",
|
||||||
|
"typescript": "^5.7.2"
|
||||||
|
},
|
||||||
|
"peerDependenciesMeta": {
|
||||||
|
"@govoplan/core-webui": {
|
||||||
|
"optional": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"files": [
|
||||||
|
"webui/src",
|
||||||
|
"README.md",
|
||||||
|
"LICENSE"
|
||||||
|
]
|
||||||
|
}
|
||||||
+2
-2
@@ -4,13 +4,13 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "govoplan-datasources"
|
name = "govoplan-datasources"
|
||||||
version = "0.1.19"
|
version = "0.1.26"
|
||||||
description = "Governed datasource catalogue, staging, and materialization lifecycle for GovOPlaN."
|
description = "Governed datasource catalogue, staging, and materialization lifecycle for GovOPlaN."
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.12"
|
requires-python = ">=3.12"
|
||||||
license = "AGPL-3.0-or-later"
|
license = "AGPL-3.0-or-later"
|
||||||
authors = [{ name = "GovOPlaN" }]
|
authors = [{ name = "GovOPlaN" }]
|
||||||
dependencies = ["govoplan-core>=0.1.18"]
|
dependencies = ["govoplan-core>=0.1.46"]
|
||||||
|
|
||||||
[tool.setuptools.packages.find]
|
[tool.setuptools.packages.find]
|
||||||
where = ["src"]
|
where = ["src"]
|
||||||
|
|||||||
@@ -1,3 +1,3 @@
|
|||||||
"""GovOPlaN Datasources module."""
|
"""GovOPlaN Datasources module."""
|
||||||
|
|
||||||
__version__ = "0.1.19"
|
__version__ = "0.1.26"
|
||||||
|
|||||||
+71
@@ -0,0 +1,71 @@
|
|||||||
|
"""Durable CSV source fidelity; no preview retention or catalogue content."""
|
||||||
|
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
|
||||||
|
from govoplan_core.core.datasources import DatasourceValidationError
|
||||||
|
from govoplan_core.core.tabular_sources import (
|
||||||
|
TabularCsvSource,
|
||||||
|
TabularSourceError,
|
||||||
|
csv_source_payload,
|
||||||
|
parse_tabular_csv,
|
||||||
|
verified_csv_source_text,
|
||||||
|
csv_projection_matches,
|
||||||
|
)
|
||||||
|
from govoplan_datasources.backend.tabular import (
|
||||||
|
MAX_STAGE_BYTES,
|
||||||
|
MAX_STAGE_ROWS,
|
||||||
|
normalize_rows,
|
||||||
|
parse_csv_rows,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def prepare_csv_source(
|
||||||
|
source: TabularCsvSource | None, rows: Sequence[Mapping[str, object]]
|
||||||
|
) -> dict[str, object] | None:
|
||||||
|
if source is None:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
payload = csv_source_payload(source, max_bytes=MAX_STAGE_BYTES)
|
||||||
|
if source.parser_profile == "core.csv.v1":
|
||||||
|
expected = parse_tabular_csv(
|
||||||
|
source.text,
|
||||||
|
delimiter=source.delimiter,
|
||||||
|
value_mode=source.value_mode,
|
||||||
|
max_rows=MAX_STAGE_ROWS,
|
||||||
|
max_bytes=MAX_STAGE_BYTES,
|
||||||
|
)
|
||||||
|
elif source.parser_profile == "datasources.csv.v1":
|
||||||
|
expected = parse_csv_rows(
|
||||||
|
source.text, delimiter=source.delimiter, value_mode=source.value_mode
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
raise DatasourceValidationError("Unsupported original CSV parser profile.")
|
||||||
|
if not csv_projection_matches(normalize_rows(expected), rows):
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"Stage rows do not match their original CSV source and parsing mode."
|
||||||
|
)
|
||||||
|
return payload
|
||||||
|
except TabularSourceError as exc:
|
||||||
|
raise DatasourceValidationError(str(exc)) from exc
|
||||||
|
|
||||||
|
|
||||||
|
def verify_stage_csv(
|
||||||
|
payload: Mapping[str, object] | None, rows: Sequence[Mapping[str, object]]
|
||||||
|
) -> None:
|
||||||
|
if payload is None:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
text = verified_csv_source_text(payload)
|
||||||
|
prepare_csv_source(
|
||||||
|
TabularCsvSource(
|
||||||
|
text=text,
|
||||||
|
delimiter=payload["delimiter"],
|
||||||
|
value_mode=payload["value_mode"],
|
||||||
|
parser_profile=payload["parser_profile"],
|
||||||
|
),
|
||||||
|
rows,
|
||||||
|
)
|
||||||
|
except (KeyError, TypeError, TabularSourceError) as exc:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"Original CSV source evidence is invalid."
|
||||||
|
) from exc
|
||||||
@@ -109,6 +109,10 @@ class DatasourceRecord(Base, TimestampMixin):
|
|||||||
)
|
)
|
||||||
privacy_profile_ref: Mapped[str | None] = mapped_column(String(500), nullable=True)
|
privacy_profile_ref: Mapped[str | None] = mapped_column(String(500), nullable=True)
|
||||||
retention_policy_ref: Mapped[str | None] = mapped_column(String(500), nullable=True)
|
retention_policy_ref: Mapped[str | None] = mapped_column(String(500), nullable=True)
|
||||||
|
access_policy_ref: Mapped[str | None] = mapped_column(String(500), nullable=True)
|
||||||
|
visibility_policy: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
hold_refs: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
|
hold_refs: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
|
||||||
publication_state: Mapped[str] = mapped_column(
|
publication_state: Mapped[str] = mapped_column(
|
||||||
String(50), default="draft", nullable=False, index=True
|
String(50), default="draft", nullable=False, index=True
|
||||||
@@ -122,6 +126,12 @@ class DatasourceRecord(Base, TimestampMixin):
|
|||||||
quality_policy: Mapped[dict[str, Any]] = mapped_column(
|
quality_policy: Mapped[dict[str, Any]] = mapped_column(
|
||||||
JSON, default=dict, nullable=False
|
JSON, default=dict, nullable=False
|
||||||
)
|
)
|
||||||
|
approval_policy: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
|
retention_policy: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
JSON, default=dict, nullable=False
|
||||||
|
)
|
||||||
known_limits: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
|
known_limits: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
|
||||||
correction_procedure_ref: Mapped[str | None] = mapped_column(
|
correction_procedure_ref: Mapped[str | None] = mapped_column(
|
||||||
String(500), nullable=True
|
String(500), nullable=True
|
||||||
@@ -235,6 +245,7 @@ class DatasourceMaterializationRecord(Base, TimestampMixin):
|
|||||||
# Kept nullable-in-practice for rolling upgrades. New materializations use
|
# Kept nullable-in-practice for rolling upgrades. New materializations use
|
||||||
# immutable payload rows and leave this legacy field empty.
|
# immutable payload rows and leave this legacy field empty.
|
||||||
rows: Mapped[list[dict[str, Any]]] = mapped_column(JSON, default=list, nullable=False)
|
rows: Mapped[list[dict[str, Any]]] = mapped_column(JSON, default=list, nullable=False)
|
||||||
|
csv_source_: Mapped[dict[str, Any] | None] = mapped_column("csv_source", JSON, nullable=True, deferred=True)
|
||||||
fingerprint: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
fingerprint: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||||
row_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
row_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||||
byte_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
byte_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||||
@@ -262,6 +273,15 @@ class DatasourceMaterializationRecord(Base, TimestampMixin):
|
|||||||
default=dict,
|
default=dict,
|
||||||
nullable=False,
|
nullable=False,
|
||||||
)
|
)
|
||||||
|
disposed_at: Mapped[datetime | None] = mapped_column(
|
||||||
|
DateTime(timezone=True), nullable=True, index=True
|
||||||
|
)
|
||||||
|
disposition_: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
"disposition",
|
||||||
|
JSON,
|
||||||
|
default=dict,
|
||||||
|
nullable=False,
|
||||||
|
)
|
||||||
created_by: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
created_by: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
||||||
|
|
||||||
datasource: Mapped[DatasourceRecord] = relationship(back_populates="materializations")
|
datasource: Mapped[DatasourceRecord] = relationship(back_populates="materializations")
|
||||||
@@ -386,6 +406,7 @@ class DatasourceStageRecord(Base, TimestampMixin):
|
|||||||
nullable=False,
|
nullable=False,
|
||||||
)
|
)
|
||||||
rows: Mapped[list[dict[str, Any]]] = mapped_column(JSON, default=list, nullable=False)
|
rows: Mapped[list[dict[str, Any]]] = mapped_column(JSON, default=list, nullable=False)
|
||||||
|
csv_source_: Mapped[dict[str, Any] | None] = mapped_column("csv_source", JSON, nullable=True, deferred=True)
|
||||||
fingerprint: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
fingerprint: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||||
row_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
row_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||||
byte_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
byte_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||||
@@ -413,6 +434,12 @@ class DatasourceStageRecord(Base, TimestampMixin):
|
|||||||
default=dict,
|
default=dict,
|
||||||
nullable=False,
|
nullable=False,
|
||||||
)
|
)
|
||||||
|
approval_: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
"approval",
|
||||||
|
JSON,
|
||||||
|
default=dict,
|
||||||
|
nullable=False,
|
||||||
|
)
|
||||||
promoted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
promoted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||||
promoted_materialization_id: Mapped[str | None] = mapped_column(
|
promoted_materialization_id: Mapped[str | None] = mapped_column(
|
||||||
String(36),
|
String(36),
|
||||||
@@ -479,7 +506,49 @@ class DatasourcePublicationRecord(Base, TimestampMixin):
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceLifecycleEvidenceRecord(Base):
|
||||||
|
__tablename__ = "datasource_lifecycle_evidence"
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"event_hash",
|
||||||
|
name="uq_datasource_lifecycle_evidence_hash",
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"ix_datasource_lifecycle_evidence_subject",
|
||||||
|
"tenant_id",
|
||||||
|
"subject_ref",
|
||||||
|
"occurred_at",
|
||||||
|
),
|
||||||
|
Index(
|
||||||
|
"ix_datasource_lifecycle_evidence_event",
|
||||||
|
"tenant_id",
|
||||||
|
"event_type",
|
||||||
|
"occurred_at",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||||
|
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||||
|
subject_ref: Mapped[str] = mapped_column(String(160), nullable=False, index=True)
|
||||||
|
event_type: Mapped[str] = mapped_column(String(80), nullable=False, index=True)
|
||||||
|
occurred_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||||
|
actor_ref: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
||||||
|
policy_version: Mapped[str | None] = mapped_column(String(120), nullable=True)
|
||||||
|
policy_hash: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||||
|
subject_digest: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||||
|
previous_event_hash: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||||
|
event_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||||
|
details_: Mapped[dict[str, Any]] = mapped_column(
|
||||||
|
"details",
|
||||||
|
JSON,
|
||||||
|
default=dict,
|
||||||
|
nullable=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
|
"DatasourceLifecycleEvidenceRecord",
|
||||||
"DatasourceMaterializationRecord",
|
"DatasourceMaterializationRecord",
|
||||||
"DatasourcePayloadRecord",
|
"DatasourcePayloadRecord",
|
||||||
"DatasourcePayloadRowRecord",
|
"DatasourcePayloadRowRecord",
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ from govoplan_core.core.dsar import (
|
|||||||
)
|
)
|
||||||
from govoplan_datasources.backend.db.models import (
|
from govoplan_datasources.backend.db.models import (
|
||||||
DatasourceGovernanceReferenceRecord,
|
DatasourceGovernanceReferenceRecord,
|
||||||
|
DatasourceLifecycleEvidenceRecord,
|
||||||
DatasourceMaterializationRecord,
|
DatasourceMaterializationRecord,
|
||||||
DatasourcePayloadRecord,
|
DatasourcePayloadRecord,
|
||||||
DatasourcePublicationRecord,
|
DatasourcePublicationRecord,
|
||||||
@@ -35,6 +36,7 @@ _DIRECT_ALIASES = {
|
|||||||
"payload_id": ("datasources.payload",),
|
"payload_id": ("datasources.payload",),
|
||||||
"stage_id": ("datasources.stage",),
|
"stage_id": ("datasources.stage",),
|
||||||
"publication_id": ("datasources.publication",),
|
"publication_id": ("datasources.publication",),
|
||||||
|
"lifecycle_evidence_id": ("datasources.lifecycle_evidence",),
|
||||||
}
|
}
|
||||||
_RESOURCE_MODELS = {
|
_RESOURCE_MODELS = {
|
||||||
"datasource": DatasourceRecord,
|
"datasource": DatasourceRecord,
|
||||||
@@ -43,6 +45,7 @@ _RESOURCE_MODELS = {
|
|||||||
"datasource_payload": DatasourcePayloadRecord,
|
"datasource_payload": DatasourcePayloadRecord,
|
||||||
"datasource_stage": DatasourceStageRecord,
|
"datasource_stage": DatasourceStageRecord,
|
||||||
"datasource_publication": DatasourcePublicationRecord,
|
"datasource_publication": DatasourcePublicationRecord,
|
||||||
|
"datasource_lifecycle_evidence": DatasourceLifecycleEvidenceRecord,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -284,6 +287,10 @@ def _direct_matches(
|
|||||||
DatasourcePublicationRecord,
|
DatasourcePublicationRecord,
|
||||||
"datasource_publication",
|
"datasource_publication",
|
||||||
),
|
),
|
||||||
|
"lifecycle_evidence_id": (
|
||||||
|
DatasourceLifecycleEvidenceRecord,
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
),
|
||||||
}[selector]
|
}[selector]
|
||||||
row = _one(
|
row = _one(
|
||||||
session,
|
session,
|
||||||
@@ -421,6 +428,11 @@ def _canonical_matches(
|
|||||||
DatasourcePublicationRecord.created_by.in_(actor_ids),
|
DatasourcePublicationRecord.created_by.in_(actor_ids),
|
||||||
"datasource_publication",
|
"datasource_publication",
|
||||||
),
|
),
|
||||||
|
(
|
||||||
|
DatasourceLifecycleEvidenceRecord,
|
||||||
|
DatasourceLifecycleEvidenceRecord.actor_ref.in_(actor_ids),
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
),
|
||||||
)
|
)
|
||||||
matches: list[_Match] = []
|
matches: list[_Match] = []
|
||||||
for model, condition, resource_type in specs:
|
for model, condition, resource_type in specs:
|
||||||
@@ -472,6 +484,8 @@ def _direct_category(session: Session, resource_type: str, row: Any) -> str:
|
|||||||
)
|
)
|
||||||
if not referenced:
|
if not referenced:
|
||||||
return "unreferenced_datasource_payload"
|
return "unreferenced_datasource_payload"
|
||||||
|
if resource_type == "datasource_lifecycle_evidence":
|
||||||
|
return "datasource_operator_attribution"
|
||||||
return {
|
return {
|
||||||
"datasource": "datasource_configuration",
|
"datasource": "datasource_configuration",
|
||||||
"datasource_governance_reference": "datasource_governance_configuration",
|
"datasource_governance_reference": "datasource_governance_configuration",
|
||||||
@@ -479,6 +493,7 @@ def _direct_category(session: Session, resource_type: str, row: Any) -> str:
|
|||||||
"datasource_payload": "referenced_datasource_payload",
|
"datasource_payload": "referenced_datasource_payload",
|
||||||
"datasource_stage": "promoted_datasource_stage",
|
"datasource_stage": "promoted_datasource_stage",
|
||||||
"datasource_publication": "immutable_datasource_publication",
|
"datasource_publication": "immutable_datasource_publication",
|
||||||
|
"datasource_lifecycle_evidence": "datasource_operator_attribution",
|
||||||
}[resource_type]
|
}[resource_type]
|
||||||
|
|
||||||
|
|
||||||
@@ -498,6 +513,22 @@ def _root_datasource_ids(session: Session, match: _Match) -> set[str]:
|
|||||||
)
|
)
|
||||||
.all()
|
.all()
|
||||||
}
|
}
|
||||||
|
if match.resource_type == "datasource_lifecycle_evidence":
|
||||||
|
if row.subject_ref.startswith("datasource:"):
|
||||||
|
return {row.subject_ref.removeprefix("datasource:")}
|
||||||
|
if row.subject_ref.startswith("stage:"):
|
||||||
|
stage = session.get(
|
||||||
|
DatasourceStageRecord,
|
||||||
|
row.subject_ref.removeprefix("stage:"),
|
||||||
|
)
|
||||||
|
return {stage.target_datasource_id} if stage and stage.target_datasource_id else set()
|
||||||
|
if row.subject_ref.startswith("materialization:"):
|
||||||
|
materialization = session.get(
|
||||||
|
DatasourceMaterializationRecord,
|
||||||
|
row.subject_ref.removeprefix("materialization:"),
|
||||||
|
)
|
||||||
|
return {materialization.datasource_id} if materialization else set()
|
||||||
|
return set()
|
||||||
return {row.datasource_id}
|
return {row.datasource_id}
|
||||||
|
|
||||||
|
|
||||||
@@ -505,7 +536,7 @@ def _correlates(match: _Match, actor_ids: tuple[str, ...]) -> bool:
|
|||||||
row = match.row
|
row = match.row
|
||||||
return any(
|
return any(
|
||||||
str(getattr(row, field, "") or "") in actor_ids
|
str(getattr(row, field, "") or "") in actor_ids
|
||||||
for field in ("created_by", "updated_by")
|
for field in ("created_by", "updated_by", "actor_ref")
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -521,6 +552,7 @@ def _directly_targets(selectors: _Selectors, match: _Match) -> bool:
|
|||||||
"datasource_payload": ("payload_id", "id"),
|
"datasource_payload": ("payload_id", "id"),
|
||||||
"datasource_stage": ("stage_id", "id"),
|
"datasource_stage": ("stage_id", "id"),
|
||||||
"datasource_publication": ("publication_id", "id"),
|
"datasource_publication": ("publication_id", "id"),
|
||||||
|
"datasource_lifecycle_evidence": ("lifecycle_evidence_id", "id"),
|
||||||
}[match.resource_type]
|
}[match.resource_type]
|
||||||
value = _strip_prefix(selectors.direct.get(selector, ""))
|
value = _strip_prefix(selectors.direct.get(selector, ""))
|
||||||
if value == str(getattr(row, field)):
|
if value == str(getattr(row, field)):
|
||||||
@@ -540,6 +572,12 @@ def _root_datasource_ids_for_row(match: _Match) -> set[str]:
|
|||||||
return {row.target_datasource_id} if row.target_datasource_id else set()
|
return {row.target_datasource_id} if row.target_datasource_id else set()
|
||||||
if match.resource_type == "datasource_payload":
|
if match.resource_type == "datasource_payload":
|
||||||
return set()
|
return set()
|
||||||
|
if match.resource_type == "datasource_lifecycle_evidence":
|
||||||
|
return (
|
||||||
|
{row.subject_ref.removeprefix("datasource:")}
|
||||||
|
if row.subject_ref.startswith("datasource:")
|
||||||
|
else set()
|
||||||
|
)
|
||||||
return {row.datasource_id}
|
return {row.datasource_id}
|
||||||
|
|
||||||
|
|
||||||
@@ -605,6 +643,13 @@ def _record_data(resource_type: str, row: Any) -> dict[str, object]:
|
|||||||
"created_at": _iso(row.created_at),
|
"created_at": _iso(row.created_at),
|
||||||
"updated_at": _iso(row.updated_at),
|
"updated_at": _iso(row.updated_at),
|
||||||
}
|
}
|
||||||
|
if resource_type == "datasource_lifecycle_evidence":
|
||||||
|
return {
|
||||||
|
"subject_ref": row.subject_ref,
|
||||||
|
"event_type": row.event_type,
|
||||||
|
"policy_version": row.policy_version,
|
||||||
|
"occurred_at": _iso(row.occurred_at),
|
||||||
|
}
|
||||||
if resource_type == "datasource_governance_reference":
|
if resource_type == "datasource_governance_reference":
|
||||||
return {"relation": row.relation}
|
return {"relation": row.relation}
|
||||||
if resource_type == "datasource_materialization":
|
if resource_type == "datasource_materialization":
|
||||||
@@ -700,7 +745,13 @@ def _title(resource_type: str) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def _observed_at(row: Any) -> datetime | None:
|
def _observed_at(row: Any) -> datetime | None:
|
||||||
for field in ("promoted_at", "source_timestamp", "updated_at", "created_at"):
|
for field in (
|
||||||
|
"promoted_at",
|
||||||
|
"source_timestamp",
|
||||||
|
"occurred_at",
|
||||||
|
"updated_at",
|
||||||
|
"created_at",
|
||||||
|
):
|
||||||
value = getattr(row, field, None)
|
value = getattr(row, field, None)
|
||||||
if isinstance(value, datetime):
|
if isinstance(value, datetime):
|
||||||
return _aware(value)
|
return _aware(value)
|
||||||
|
|||||||
@@ -0,0 +1,75 @@
|
|||||||
|
"""German translations for public structured documentation metadata."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
GERMAN_STRUCTURED_TRANSLATIONS: dict[str, dict[str, Any]] = {'datasources.approval-and-retention': {'limitations': ['Die Ausführung der Retention ist explizit '
|
||||||
|
'und plangebunden; dieses Modul führt '
|
||||||
|
'keinen versteckten Löschplan aus.',
|
||||||
|
'Disponierte '
|
||||||
|
'Materialisierungsaufzeichnungen behalten '
|
||||||
|
'minimierte Schema- und Herkunftsmetadaten '
|
||||||
|
'bei, während ihre Nutzdatenzeilen '
|
||||||
|
'entfernt werden.']},
|
||||||
|
'datasources.quality-gates': {'limitations': ['Referenzregeln verwenden derzeit einen begrenzten '
|
||||||
|
'eingebetteten Wertsatz, anstatt eine andere '
|
||||||
|
'geschützte Datenquelle zu lesen.',
|
||||||
|
'Genehmigungen sind modullokale '
|
||||||
|
'Lebenszyklusnachweise; die Installation eines '
|
||||||
|
'separaten Genehmigungsmoduls ändert nicht '
|
||||||
|
'stillschweigend die konfigurierte '
|
||||||
|
'Datenquellenrichtlinie.',
|
||||||
|
'Die Aufbewahrung erfolgt nie automatisch: Ein '
|
||||||
|
'Administrator muss eine Vorschau eines aktuellen '
|
||||||
|
'Plans anzeigen und ausdrücklich berechtigte Ziele '
|
||||||
|
'anwenden.',
|
||||||
|
'Artefaktbytes bleiben im Besitz ihres '
|
||||||
|
'Payload-Backends; Datasources speichert einen '
|
||||||
|
'unveränderlichen Referenz- und '
|
||||||
|
'Integritätsnachweis.']},
|
||||||
|
'datasources.reference.fields-and-consequences': {'consequence_classes': {'freeze': 'Erstellt '
|
||||||
|
'einen '
|
||||||
|
'gekennzeichneten '
|
||||||
|
'unveränderlichen '
|
||||||
|
'Zustand für '
|
||||||
|
'reproduzierbare '
|
||||||
|
'Verbraucher '
|
||||||
|
'und '
|
||||||
|
'Nachweise.',
|
||||||
|
'promote': 'Erstellt '
|
||||||
|
'oder '
|
||||||
|
'aktualisiert '
|
||||||
|
'eine '
|
||||||
|
'geregelte '
|
||||||
|
'Datenquelle '
|
||||||
|
'und fügt '
|
||||||
|
'eine '
|
||||||
|
'unveränderliche '
|
||||||
|
'Materialisierungsrevision '
|
||||||
|
'hinzu.',
|
||||||
|
'refresh': 'Leset den '
|
||||||
|
'zwischengespeicherten '
|
||||||
|
'Ursprung '
|
||||||
|
'und fügt '
|
||||||
|
'eine neue '
|
||||||
|
'unveränderliche '
|
||||||
|
'Materialisierungsrevision '
|
||||||
|
'hinzu.',
|
||||||
|
'retire': 'Verhindert '
|
||||||
|
'eine neue '
|
||||||
|
'Auswahl, '
|
||||||
|
'während die '
|
||||||
|
'Referenzen '
|
||||||
|
'für die '
|
||||||
|
'Materialisierung '
|
||||||
|
'beibehalten '
|
||||||
|
'werden.'}},
|
||||||
|
'datasources.visibility': {'limitations': ['Zeilenfilter unterstützen deterministische Gleichheit '
|
||||||
|
'oder Mitgliedschaft gegen Hauptansprüche; '
|
||||||
|
'willkürliche Ausdrücke werden nicht ausgeführt.',
|
||||||
|
'Live-gefilterte Vorschauen scannen maximal 10.000 '
|
||||||
|
'Ursprungszeilen und legen eine strukturierte '
|
||||||
|
'Einschränkungsdiagnose offen, wenn der Ursprung '
|
||||||
|
'größer ist.']}}
|
||||||
@@ -0,0 +1,723 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
|
||||||
|
from sqlalchemy import func, select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_core.core.datasources import DatasourceValidationError
|
||||||
|
from govoplan_core.db.base import utcnow
|
||||||
|
from govoplan_datasources.backend.db.models import (
|
||||||
|
DatasourceLifecycleEvidenceRecord,
|
||||||
|
DatasourceMaterializationRecord,
|
||||||
|
DatasourcePublicationRecord,
|
||||||
|
DatasourceRecord,
|
||||||
|
DatasourceStageRecord,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class RetentionCandidate:
|
||||||
|
ref: str
|
||||||
|
kind: str
|
||||||
|
datasource_ref: str | None
|
||||||
|
disposition: str
|
||||||
|
eligible_at: datetime
|
||||||
|
eligible: bool
|
||||||
|
blockers: tuple[str, ...]
|
||||||
|
policy_version: str
|
||||||
|
policy_hash: str
|
||||||
|
|
||||||
|
def to_dict(self) -> dict[str, object]:
|
||||||
|
return {
|
||||||
|
"ref": self.ref,
|
||||||
|
"kind": self.kind,
|
||||||
|
"datasource_ref": self.datasource_ref,
|
||||||
|
"disposition": self.disposition,
|
||||||
|
"eligible_at": _as_utc(self.eligible_at).isoformat(),
|
||||||
|
"eligible": self.eligible,
|
||||||
|
"blockers": list(self.blockers),
|
||||||
|
"policy_version": self.policy_version,
|
||||||
|
"policy_hash": self.policy_hash,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class RetentionPlan:
|
||||||
|
as_of: datetime
|
||||||
|
plan_hash: str
|
||||||
|
candidates: tuple[RetentionCandidate, ...]
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_hash(value: object) -> str:
|
||||||
|
encoded = json.dumps(
|
||||||
|
value,
|
||||||
|
sort_keys=True,
|
||||||
|
separators=(",", ":"),
|
||||||
|
ensure_ascii=True,
|
||||||
|
default=str,
|
||||||
|
)
|
||||||
|
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_approval_policy(value: Mapping[str, object] | None) -> dict[str, object]:
|
||||||
|
source = dict(value or {})
|
||||||
|
version = str(source.get("version") or "1").strip()
|
||||||
|
if not version:
|
||||||
|
raise DatasourceValidationError("Approval policy version is required.")
|
||||||
|
required = _boolean(source.get("required", False), label="required")
|
||||||
|
required_approvals = _bounded_integer(
|
||||||
|
source.get("required_approvals", 1),
|
||||||
|
label="required_approvals",
|
||||||
|
minimum=1,
|
||||||
|
maximum=5,
|
||||||
|
)
|
||||||
|
separation = _boolean(
|
||||||
|
source.get("separation_of_duties", True),
|
||||||
|
label="separation_of_duties",
|
||||||
|
)
|
||||||
|
expires_after_hours = source.get("expires_after_hours")
|
||||||
|
expiry = (
|
||||||
|
None
|
||||||
|
if expires_after_hours is None
|
||||||
|
else _bounded_integer(
|
||||||
|
expires_after_hours,
|
||||||
|
label="expires_after_hours",
|
||||||
|
minimum=1,
|
||||||
|
maximum=8_760,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
policy_ref = _optional_text(source.get("policy_ref"))
|
||||||
|
return {
|
||||||
|
"version": version,
|
||||||
|
"required": required,
|
||||||
|
"required_approvals": required_approvals,
|
||||||
|
"separation_of_duties": separation,
|
||||||
|
"expires_after_hours": expiry,
|
||||||
|
"policy_ref": policy_ref,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_retention_policy(value: Mapping[str, object] | None) -> dict[str, object]:
|
||||||
|
source = dict(value or {})
|
||||||
|
version = str(source.get("version") or "1").strip()
|
||||||
|
if not version:
|
||||||
|
raise DatasourceValidationError("Retention policy version is required.")
|
||||||
|
enabled = _boolean(source.get("enabled", False), label="enabled")
|
||||||
|
durations = {
|
||||||
|
key: _optional_duration(source.get(key), label=key)
|
||||||
|
for key in (
|
||||||
|
"stage_days",
|
||||||
|
"materialization_days",
|
||||||
|
"frozen_evidence_days",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
if enabled and all(value is None for value in durations.values()):
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"Enabled retention policy requires at least one retention duration."
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"version": version,
|
||||||
|
"enabled": enabled,
|
||||||
|
**durations,
|
||||||
|
"policy_ref": _optional_text(source.get("policy_ref")),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def initialize_stage_approval(
|
||||||
|
stage: DatasourceStageRecord,
|
||||||
|
*,
|
||||||
|
created_at: datetime | None = None,
|
||||||
|
) -> dict[str, object]:
|
||||||
|
policy = normalize_approval_policy(
|
||||||
|
_mapping(stage.governance_).get("approval_policy")
|
||||||
|
if isinstance(_mapping(stage.governance_).get("approval_policy"), Mapping)
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
now = _as_utc(created_at or utcnow())
|
||||||
|
expiry_hours = policy["expires_after_hours"]
|
||||||
|
expires_at = (
|
||||||
|
now + timedelta(hours=int(expiry_hours))
|
||||||
|
if isinstance(expiry_hours, int)
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
approval = {
|
||||||
|
"state": "pending" if policy["required"] else "not_required",
|
||||||
|
"policy": policy,
|
||||||
|
"policy_hash": canonical_hash(policy),
|
||||||
|
"subject_digest": stage_subject_digest(stage, policy=policy),
|
||||||
|
"required_approvals": policy["required_approvals"],
|
||||||
|
"approval_count": 0,
|
||||||
|
"expires_at": expires_at.isoformat() if expires_at else None,
|
||||||
|
"approvals": [],
|
||||||
|
}
|
||||||
|
stage.approval_ = approval
|
||||||
|
if stage.validation_.get("valid") is True:
|
||||||
|
stage.state = "awaiting_approval" if policy["required"] else "ready"
|
||||||
|
return approval
|
||||||
|
|
||||||
|
|
||||||
|
def decide_stage(
|
||||||
|
stage: DatasourceStageRecord,
|
||||||
|
*,
|
||||||
|
actor_ref: str,
|
||||||
|
actor_scopes: Sequence[str],
|
||||||
|
decision: str,
|
||||||
|
reason: str,
|
||||||
|
expected_policy_hash: str,
|
||||||
|
expected_subject_digest: str,
|
||||||
|
now: datetime | None = None,
|
||||||
|
) -> tuple[dict[str, object], bool]:
|
||||||
|
approval = dict(stage.approval_ or {})
|
||||||
|
if approval.get("policy_hash") != expected_policy_hash:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"The approval policy changed; reload the stage before deciding."
|
||||||
|
)
|
||||||
|
if approval.get("subject_digest") != expected_subject_digest:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"The staged content changed; reload the stage before deciding."
|
||||||
|
)
|
||||||
|
policy = normalize_approval_policy(_mapping(approval.get("policy")))
|
||||||
|
if not policy["required"]:
|
||||||
|
raise DatasourceValidationError("This stage does not require approval.")
|
||||||
|
if stage.state not in {"awaiting_approval", "ready"}:
|
||||||
|
raise DatasourceValidationError("This stage no longer accepts approval decisions.")
|
||||||
|
if policy["separation_of_duties"] and stage.created_by == actor_ref:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"The stage creator cannot approve this stage under separation of duties."
|
||||||
|
)
|
||||||
|
decided_at = _as_utc(now or utcnow())
|
||||||
|
expires_at = _optional_datetime(approval.get("expires_at"))
|
||||||
|
if expires_at is not None and decided_at > expires_at:
|
||||||
|
approval["state"] = "expired"
|
||||||
|
stage.approval_ = approval
|
||||||
|
stage.state = "awaiting_approval"
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"The approval window expired; create a new stage for a fresh decision."
|
||||||
|
)
|
||||||
|
decisions = [dict(item) for item in _mapping_sequence(approval.get("approvals"))]
|
||||||
|
existing = next(
|
||||||
|
(item for item in decisions if item.get("actor_ref") == actor_ref),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if existing is not None:
|
||||||
|
if existing.get("decision") == decision:
|
||||||
|
return approval, True
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"This approver already recorded a different decision."
|
||||||
|
)
|
||||||
|
cleaned_reason = reason.strip()
|
||||||
|
if not cleaned_reason:
|
||||||
|
raise DatasourceValidationError("An approval decision reason is required.")
|
||||||
|
decisions.append(
|
||||||
|
{
|
||||||
|
"actor_ref": actor_ref,
|
||||||
|
"decision": decision,
|
||||||
|
"reason": cleaned_reason,
|
||||||
|
"decided_at": decided_at.isoformat(),
|
||||||
|
"authority_scopes": sorted(set(actor_scopes)),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
approved = sum(item.get("decision") == "approve" for item in decisions)
|
||||||
|
approval.update(
|
||||||
|
{
|
||||||
|
"approvals": decisions,
|
||||||
|
"approval_count": approved,
|
||||||
|
"state": (
|
||||||
|
"rejected"
|
||||||
|
if decision == "reject"
|
||||||
|
else (
|
||||||
|
"approved"
|
||||||
|
if approved >= int(policy["required_approvals"])
|
||||||
|
else "pending"
|
||||||
|
)
|
||||||
|
),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
stage.approval_ = approval
|
||||||
|
stage.state = "rejected" if decision == "reject" else (
|
||||||
|
"ready"
|
||||||
|
if approval["state"] == "approved"
|
||||||
|
else "awaiting_approval"
|
||||||
|
)
|
||||||
|
return approval, False
|
||||||
|
|
||||||
|
|
||||||
|
def ensure_stage_approval_current(
|
||||||
|
stage: DatasourceStageRecord,
|
||||||
|
*,
|
||||||
|
current_policy: Mapping[str, object] | None,
|
||||||
|
) -> None:
|
||||||
|
approval = _mapping(stage.approval_)
|
||||||
|
stage_policy = normalize_approval_policy(_mapping(approval.get("policy")))
|
||||||
|
effective = normalize_approval_policy(current_policy)
|
||||||
|
if canonical_hash(effective) != approval.get("policy_hash"):
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"The approval policy changed after staging; create a new stage."
|
||||||
|
)
|
||||||
|
if stage_policy["required"] and approval.get("state") != "approved":
|
||||||
|
raise DatasourceValidationError("The stage has not reached its approval quorum.")
|
||||||
|
if approval.get("subject_digest") != stage_subject_digest(stage, policy=stage_policy):
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"The staged evidence changed after approval; create a new stage."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def stage_subject_digest(
|
||||||
|
stage: DatasourceStageRecord,
|
||||||
|
*,
|
||||||
|
policy: Mapping[str, object] | None = None,
|
||||||
|
) -> str:
|
||||||
|
return canonical_hash(
|
||||||
|
{
|
||||||
|
"stage_ref": _stage_ref(stage.id),
|
||||||
|
"target_datasource_ref": (
|
||||||
|
_datasource_ref(stage.target_datasource_id)
|
||||||
|
if stage.target_datasource_id
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
"source_name": stage.source_name,
|
||||||
|
"mode": stage.mode,
|
||||||
|
"shape": stage.shape,
|
||||||
|
"fingerprint": stage.fingerprint,
|
||||||
|
"validation_policy_hash": stage.validation_.get("policy_hash"),
|
||||||
|
"approval_policy": dict(policy or {}),
|
||||||
|
**({"csv_source": {
|
||||||
|
key: stage.csv_source_.get(key)
|
||||||
|
for key in ("sha256", "byte_count", "delimiter", "value_mode", "parser_profile")
|
||||||
|
}} if stage.csv_source_ is not None else {}),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def record_lifecycle_evidence(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
subject_ref: str,
|
||||||
|
event_type: str,
|
||||||
|
actor_ref: str | None,
|
||||||
|
subject_digest: str,
|
||||||
|
details: Mapping[str, object],
|
||||||
|
policy_version: str | None = None,
|
||||||
|
policy_hash: str | None = None,
|
||||||
|
occurred_at: datetime | None = None,
|
||||||
|
) -> DatasourceLifecycleEvidenceRecord:
|
||||||
|
happened = _as_utc(occurred_at or utcnow())
|
||||||
|
previous_hash = session.scalar(
|
||||||
|
select(DatasourceLifecycleEvidenceRecord.event_hash)
|
||||||
|
.where(
|
||||||
|
DatasourceLifecycleEvidenceRecord.tenant_id == tenant_id,
|
||||||
|
DatasourceLifecycleEvidenceRecord.subject_ref == subject_ref,
|
||||||
|
)
|
||||||
|
.order_by(
|
||||||
|
DatasourceLifecycleEvidenceRecord.occurred_at.desc(),
|
||||||
|
DatasourceLifecycleEvidenceRecord.id.desc(),
|
||||||
|
)
|
||||||
|
.limit(1)
|
||||||
|
)
|
||||||
|
payload = {
|
||||||
|
"tenant_id": tenant_id,
|
||||||
|
"subject_ref": subject_ref,
|
||||||
|
"event_type": event_type,
|
||||||
|
"occurred_at": happened.isoformat(),
|
||||||
|
"actor_ref": actor_ref,
|
||||||
|
"policy_version": policy_version,
|
||||||
|
"policy_hash": policy_hash,
|
||||||
|
"subject_digest": subject_digest,
|
||||||
|
"previous_event_hash": previous_hash,
|
||||||
|
"details": dict(details),
|
||||||
|
}
|
||||||
|
item = DatasourceLifecycleEvidenceRecord(
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
subject_ref=subject_ref,
|
||||||
|
event_type=event_type,
|
||||||
|
occurred_at=happened,
|
||||||
|
actor_ref=actor_ref,
|
||||||
|
policy_version=policy_version,
|
||||||
|
policy_hash=policy_hash,
|
||||||
|
subject_digest=subject_digest,
|
||||||
|
previous_event_hash=previous_hash,
|
||||||
|
event_hash=canonical_hash(payload),
|
||||||
|
details_=dict(details),
|
||||||
|
)
|
||||||
|
session.add(item)
|
||||||
|
session.flush()
|
||||||
|
return item
|
||||||
|
|
||||||
|
|
||||||
|
def list_lifecycle_evidence(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
subject_ref: str | None = None,
|
||||||
|
limit: int = 200,
|
||||||
|
) -> tuple[DatasourceLifecycleEvidenceRecord, ...]:
|
||||||
|
statement = select(DatasourceLifecycleEvidenceRecord).where(
|
||||||
|
DatasourceLifecycleEvidenceRecord.tenant_id == tenant_id
|
||||||
|
)
|
||||||
|
if subject_ref:
|
||||||
|
statement = statement.where(
|
||||||
|
DatasourceLifecycleEvidenceRecord.subject_ref == subject_ref
|
||||||
|
)
|
||||||
|
statement = statement.order_by(
|
||||||
|
DatasourceLifecycleEvidenceRecord.occurred_at.desc(),
|
||||||
|
DatasourceLifecycleEvidenceRecord.id.desc(),
|
||||||
|
).limit(max(1, min(limit, 500)))
|
||||||
|
return tuple(session.scalars(statement))
|
||||||
|
|
||||||
|
|
||||||
|
def build_retention_plan(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
as_of: datetime,
|
||||||
|
) -> RetentionPlan:
|
||||||
|
effective_at = _as_utc(as_of)
|
||||||
|
candidates: list[RetentionCandidate] = []
|
||||||
|
stages = session.scalars(
|
||||||
|
select(DatasourceStageRecord).where(
|
||||||
|
DatasourceStageRecord.tenant_id == tenant_id
|
||||||
|
)
|
||||||
|
)
|
||||||
|
for stage in stages:
|
||||||
|
policy = normalize_retention_policy(
|
||||||
|
_mapping(_mapping(stage.governance_).get("retention_policy"))
|
||||||
|
)
|
||||||
|
duration = policy.get("stage_days")
|
||||||
|
if not policy["enabled"] or not isinstance(duration, int):
|
||||||
|
continue
|
||||||
|
eligible_at = _as_utc(stage.created_at) + timedelta(days=duration)
|
||||||
|
if eligible_at > effective_at:
|
||||||
|
continue
|
||||||
|
blockers = (
|
||||||
|
("pending_approval",)
|
||||||
|
if stage.state == "awaiting_approval"
|
||||||
|
else ()
|
||||||
|
)
|
||||||
|
candidates.append(
|
||||||
|
RetentionCandidate(
|
||||||
|
ref=_stage_ref(stage.id),
|
||||||
|
kind="stage",
|
||||||
|
datasource_ref=(
|
||||||
|
_datasource_ref(stage.target_datasource_id)
|
||||||
|
if stage.target_datasource_id
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
disposition="delete_stage",
|
||||||
|
eligible_at=eligible_at,
|
||||||
|
eligible=not blockers,
|
||||||
|
blockers=blockers,
|
||||||
|
policy_version=str(policy["version"]),
|
||||||
|
policy_hash=canonical_hash(policy),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
materializations = session.execute(
|
||||||
|
select(DatasourceMaterializationRecord, DatasourceRecord)
|
||||||
|
.join(DatasourceRecord, DatasourceRecord.id == DatasourceMaterializationRecord.datasource_id)
|
||||||
|
.where(DatasourceMaterializationRecord.tenant_id == tenant_id)
|
||||||
|
)
|
||||||
|
for materialization, datasource in materializations:
|
||||||
|
if materialization.disposed_at is not None:
|
||||||
|
continue
|
||||||
|
snapshot = _mapping(materialization.governance_snapshot_)
|
||||||
|
policy = normalize_retention_policy(
|
||||||
|
_mapping(snapshot.get("retention_policy"))
|
||||||
|
)
|
||||||
|
duration_key = (
|
||||||
|
"frozen_evidence_days"
|
||||||
|
if materialization.frozen_at is not None
|
||||||
|
else "materialization_days"
|
||||||
|
)
|
||||||
|
duration = policy.get(duration_key)
|
||||||
|
if not policy["enabled"] or not isinstance(duration, int):
|
||||||
|
continue
|
||||||
|
eligible_at = _as_utc(materialization.created_at) + timedelta(days=duration)
|
||||||
|
if eligible_at > effective_at:
|
||||||
|
continue
|
||||||
|
blockers: list[str] = []
|
||||||
|
if datasource.current_materialization_id == materialization.id:
|
||||||
|
blockers.append("current_materialization")
|
||||||
|
if tuple(snapshot.get("hold_refs") or ()):
|
||||||
|
blockers.append("legal_hold")
|
||||||
|
publication_count = session.scalar(
|
||||||
|
select(func.count(DatasourcePublicationRecord.id)).where(
|
||||||
|
DatasourcePublicationRecord.materialization_id == materialization.id
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if publication_count:
|
||||||
|
blockers.append("publication_evidence")
|
||||||
|
candidates.append(
|
||||||
|
RetentionCandidate(
|
||||||
|
ref=_materialization_ref(materialization.id),
|
||||||
|
kind="materialization",
|
||||||
|
datasource_ref=_datasource_ref(datasource.id),
|
||||||
|
disposition="purge_materialization_payload",
|
||||||
|
eligible_at=eligible_at,
|
||||||
|
eligible=not blockers,
|
||||||
|
blockers=tuple(blockers),
|
||||||
|
policy_version=str(policy["version"]),
|
||||||
|
policy_hash=canonical_hash(policy),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
ordered = tuple(sorted(candidates, key=lambda item: (item.kind, item.ref)))
|
||||||
|
plan_payload = {
|
||||||
|
"tenant_id": tenant_id,
|
||||||
|
"as_of": effective_at.isoformat(),
|
||||||
|
"candidates": [item.to_dict() for item in ordered],
|
||||||
|
}
|
||||||
|
return RetentionPlan(
|
||||||
|
as_of=effective_at,
|
||||||
|
plan_hash=canonical_hash(plan_payload),
|
||||||
|
candidates=ordered,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def apply_retention_plan(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
actor_ref: str,
|
||||||
|
plan: RetentionPlan,
|
||||||
|
target_refs: Sequence[str],
|
||||||
|
) -> tuple[tuple[str, ...], tuple[str, ...]]:
|
||||||
|
by_ref = {item.ref: item for item in plan.candidates}
|
||||||
|
requested = tuple(dict.fromkeys(target_refs))
|
||||||
|
missing = [ref for ref in requested if ref not in by_ref]
|
||||||
|
if missing:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Retention targets are not in the current plan: {', '.join(missing)}."
|
||||||
|
)
|
||||||
|
blocked = [ref for ref in requested if not by_ref[ref].eligible]
|
||||||
|
if blocked:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Retention targets have active blockers: {', '.join(blocked)}."
|
||||||
|
)
|
||||||
|
disposed: list[str] = []
|
||||||
|
evidence_hashes: list[str] = []
|
||||||
|
for ref in requested:
|
||||||
|
candidate = by_ref[ref]
|
||||||
|
if candidate.kind == "stage":
|
||||||
|
stage = _stage_by_ref(session, tenant_id=tenant_id, ref=ref)
|
||||||
|
digest = stage_subject_digest(
|
||||||
|
stage,
|
||||||
|
policy=normalize_approval_policy(
|
||||||
|
_mapping(_mapping(stage.approval_).get("policy"))
|
||||||
|
),
|
||||||
|
)
|
||||||
|
evidence = record_lifecycle_evidence(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
subject_ref=ref,
|
||||||
|
event_type="retention.stage_deleted",
|
||||||
|
actor_ref=actor_ref,
|
||||||
|
subject_digest=digest,
|
||||||
|
policy_version=candidate.policy_version,
|
||||||
|
policy_hash=candidate.policy_hash,
|
||||||
|
details={
|
||||||
|
"plan_hash": plan.plan_hash,
|
||||||
|
"disposition": candidate.disposition,
|
||||||
|
"validation_policy_hash": stage.validation_.get("policy_hash"),
|
||||||
|
"approval_policy_hash": stage.approval_.get("policy_hash"),
|
||||||
|
},
|
||||||
|
occurred_at=plan.as_of,
|
||||||
|
)
|
||||||
|
evidence_hashes.append(evidence.event_hash)
|
||||||
|
session.delete(stage)
|
||||||
|
else:
|
||||||
|
materialization = _materialization_by_ref(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
ref=ref,
|
||||||
|
)
|
||||||
|
payload = materialization.payload
|
||||||
|
subject_digest = canonical_hash(
|
||||||
|
{
|
||||||
|
"ref": ref,
|
||||||
|
"fingerprint": materialization.fingerprint,
|
||||||
|
"payload_checksum": materialization.payload_checksum,
|
||||||
|
"revision": materialization.revision,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
evidence = record_lifecycle_evidence(
|
||||||
|
session,
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
subject_ref=ref,
|
||||||
|
event_type="retention.materialization_payload_purged",
|
||||||
|
actor_ref=actor_ref,
|
||||||
|
subject_digest=subject_digest,
|
||||||
|
policy_version=candidate.policy_version,
|
||||||
|
policy_hash=candidate.policy_hash,
|
||||||
|
details={
|
||||||
|
"plan_hash": plan.plan_hash,
|
||||||
|
"disposition": candidate.disposition,
|
||||||
|
"payload_checksum": materialization.payload_checksum,
|
||||||
|
"row_count": materialization.row_count,
|
||||||
|
"byte_count": materialization.byte_count,
|
||||||
|
"frozen": materialization.frozen_at is not None,
|
||||||
|
},
|
||||||
|
occurred_at=plan.as_of,
|
||||||
|
)
|
||||||
|
evidence_hashes.append(evidence.event_hash)
|
||||||
|
materialization.payload_id = None
|
||||||
|
materialization.rows = []
|
||||||
|
materialization.csv_source_ = None
|
||||||
|
materialization.state = "disposed"
|
||||||
|
materialization.disposed_at = plan.as_of
|
||||||
|
materialization.disposition_ = {
|
||||||
|
"plan_hash": plan.plan_hash,
|
||||||
|
"policy_version": candidate.policy_version,
|
||||||
|
"policy_hash": candidate.policy_hash,
|
||||||
|
"actor_ref": actor_ref,
|
||||||
|
"evidence_hash": evidence.event_hash,
|
||||||
|
"payload_checksum": materialization.payload_checksum,
|
||||||
|
}
|
||||||
|
session.flush()
|
||||||
|
if payload is not None:
|
||||||
|
remaining = session.scalar(
|
||||||
|
select(func.count(DatasourceMaterializationRecord.id)).where(
|
||||||
|
DatasourceMaterializationRecord.payload_id == payload.id
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if not remaining:
|
||||||
|
session.delete(payload)
|
||||||
|
disposed.append(ref)
|
||||||
|
session.flush()
|
||||||
|
return tuple(disposed), tuple(evidence_hashes)
|
||||||
|
|
||||||
|
|
||||||
|
def _stage_by_ref(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
ref: str,
|
||||||
|
) -> DatasourceStageRecord:
|
||||||
|
item = session.scalar(
|
||||||
|
select(DatasourceStageRecord).where(
|
||||||
|
DatasourceStageRecord.id == _ref_id(ref, "stage"),
|
||||||
|
DatasourceStageRecord.tenant_id == tenant_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if item is None:
|
||||||
|
raise DatasourceValidationError("Datasource stage is no longer available.")
|
||||||
|
return item
|
||||||
|
|
||||||
|
|
||||||
|
def _materialization_by_ref(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
tenant_id: str,
|
||||||
|
ref: str,
|
||||||
|
) -> DatasourceMaterializationRecord:
|
||||||
|
item = session.scalar(
|
||||||
|
select(DatasourceMaterializationRecord).where(
|
||||||
|
DatasourceMaterializationRecord.id == _ref_id(ref, "materialization"),
|
||||||
|
DatasourceMaterializationRecord.tenant_id == tenant_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if item is None:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"Datasource materialization is no longer available."
|
||||||
|
)
|
||||||
|
return item
|
||||||
|
|
||||||
|
|
||||||
|
def _mapping(value: object) -> Mapping[str, object]:
|
||||||
|
return value if isinstance(value, Mapping) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def _mapping_sequence(value: object) -> tuple[Mapping[str, object], ...]:
|
||||||
|
if not isinstance(value, Sequence) or isinstance(value, (str, bytes)):
|
||||||
|
return ()
|
||||||
|
return tuple(item for item in value if isinstance(item, Mapping))
|
||||||
|
|
||||||
|
|
||||||
|
def _boolean(value: object, *, label: str) -> bool:
|
||||||
|
if not isinstance(value, bool):
|
||||||
|
raise DatasourceValidationError(f"Approval or retention {label} must be boolean.")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _bounded_integer(
|
||||||
|
value: object,
|
||||||
|
*,
|
||||||
|
label: str,
|
||||||
|
minimum: int,
|
||||||
|
maximum: int,
|
||||||
|
) -> int:
|
||||||
|
if isinstance(value, bool) or not isinstance(value, int):
|
||||||
|
raise DatasourceValidationError(f"Policy {label} must be an integer.")
|
||||||
|
if value < minimum or value > maximum:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Policy {label} must be between {minimum} and {maximum}."
|
||||||
|
)
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _optional_duration(value: object, *, label: str) -> int | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
return _bounded_integer(value, label=label, minimum=1, maximum=36_500)
|
||||||
|
|
||||||
|
|
||||||
|
def _optional_text(value: object) -> str | None:
|
||||||
|
text = str(value or "").strip()
|
||||||
|
return text or None
|
||||||
|
|
||||||
|
|
||||||
|
def _optional_datetime(value: object) -> datetime | None:
|
||||||
|
if not value:
|
||||||
|
return None
|
||||||
|
if isinstance(value, datetime):
|
||||||
|
return _as_utc(value)
|
||||||
|
try:
|
||||||
|
return _as_utc(datetime.fromisoformat(str(value)))
|
||||||
|
except ValueError as exc:
|
||||||
|
raise DatasourceValidationError("Approval expiry is invalid.") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _as_utc(value: datetime) -> datetime:
|
||||||
|
if value.tzinfo is None:
|
||||||
|
return value.replace(tzinfo=UTC)
|
||||||
|
return value.astimezone(UTC)
|
||||||
|
|
||||||
|
|
||||||
|
def _ref_id(ref: str, prefix: str) -> str:
|
||||||
|
expected = f"{prefix}:"
|
||||||
|
if not ref.startswith(expected) or not ref.removeprefix(expected).strip():
|
||||||
|
raise DatasourceValidationError(f"Invalid {prefix} reference.")
|
||||||
|
return ref.removeprefix(expected)
|
||||||
|
|
||||||
|
|
||||||
|
def _stage_ref(value: str) -> str:
|
||||||
|
return f"stage:{value}"
|
||||||
|
|
||||||
|
|
||||||
|
def _datasource_ref(value: str) -> str:
|
||||||
|
return f"datasource:{value}"
|
||||||
|
|
||||||
|
|
||||||
|
def _materialization_ref(value: str) -> str:
|
||||||
|
return f"materialization:{value}"
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"RetentionCandidate",
|
||||||
|
"RetentionPlan",
|
||||||
|
"apply_retention_plan",
|
||||||
|
"build_retention_plan",
|
||||||
|
"canonical_hash",
|
||||||
|
"decide_stage",
|
||||||
|
"ensure_stage_approval_current",
|
||||||
|
"initialize_stage_approval",
|
||||||
|
"list_lifecycle_evidence",
|
||||||
|
"normalize_approval_policy",
|
||||||
|
"normalize_retention_policy",
|
||||||
|
"record_lifecycle_evidence",
|
||||||
|
"stage_subject_digest",
|
||||||
|
]
|
||||||
@@ -1,5 +1,8 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from govoplan_core.core.modules import with_documentation_structured_translations
|
||||||
|
from govoplan_datasources.backend.german_structured_documentation import GERMAN_STRUCTURED_TRANSLATIONS
|
||||||
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from govoplan_core.core.access import (
|
from govoplan_core.core.access import (
|
||||||
@@ -12,6 +15,7 @@ from govoplan_core.core.datasources import (
|
|||||||
CAPABILITY_DATASOURCE_LIFECYCLE,
|
CAPABILITY_DATASOURCE_LIFECYCLE,
|
||||||
CAPABILITY_DATASOURCE_ORIGINS,
|
CAPABILITY_DATASOURCE_ORIGINS,
|
||||||
CAPABILITY_DATASOURCE_PUBLICATION,
|
CAPABILITY_DATASOURCE_PUBLICATION,
|
||||||
|
CAPABILITY_POLICY_DATASOURCE_VISIBILITY,
|
||||||
datasource_artifact_backend_provider,
|
datasource_artifact_backend_provider,
|
||||||
)
|
)
|
||||||
from govoplan_core.core.module_guards import (
|
from govoplan_core.core.module_guards import (
|
||||||
@@ -20,6 +24,7 @@ from govoplan_core.core.module_guards import (
|
|||||||
)
|
)
|
||||||
from govoplan_core.core.modules import (
|
from govoplan_core.core.modules import (
|
||||||
CapabilityDocumentation,
|
CapabilityDocumentation,
|
||||||
|
DocumentationCondition,
|
||||||
DocumentationLink,
|
DocumentationLink,
|
||||||
DocumentationTopic,
|
DocumentationTopic,
|
||||||
FrontendModule,
|
FrontendModule,
|
||||||
@@ -54,6 +59,7 @@ from govoplan_datasources.backend.service import (
|
|||||||
ADMIN_SCOPE,
|
ADMIN_SCOPE,
|
||||||
CATALOGUE_READ_SCOPE,
|
CATALOGUE_READ_SCOPE,
|
||||||
SOURCE_WRITE_SCOPE,
|
SOURCE_WRITE_SCOPE,
|
||||||
|
STAGE_APPROVE_SCOPE,
|
||||||
STAGE_WRITE_SCOPE,
|
STAGE_WRITE_SCOPE,
|
||||||
SqlDatasourceProvider,
|
SqlDatasourceProvider,
|
||||||
)
|
)
|
||||||
@@ -62,7 +68,7 @@ from govoplan_datasources.backend.payloads import ExternalArtifactPayloadBackend
|
|||||||
|
|
||||||
MODULE_ID = "datasources"
|
MODULE_ID = "datasources"
|
||||||
MODULE_NAME = "Datasources"
|
MODULE_NAME = "Datasources"
|
||||||
MODULE_VERSION = "0.1.19"
|
MODULE_VERSION = "0.1.26"
|
||||||
DATASOURCE_INTERFACE_VERSION = "0.2.0"
|
DATASOURCE_INTERFACE_VERSION = "0.2.0"
|
||||||
|
|
||||||
ARCHITECTURE = ModuleArchitectureDeclaration(
|
ARCHITECTURE = ModuleArchitectureDeclaration(
|
||||||
@@ -148,6 +154,11 @@ PERMISSIONS = (
|
|||||||
"Stage datasource content",
|
"Stage datasource content",
|
||||||
"Upload, validate, inspect, and promote bounded datasource stages.",
|
"Upload, validate, inspect, and promote bounded datasource stages.",
|
||||||
),
|
),
|
||||||
|
_permission(
|
||||||
|
STAGE_APPROVE_SCOPE,
|
||||||
|
"Approve datasource promotion",
|
||||||
|
"Review staged validation evidence and record an attributable promotion decision.",
|
||||||
|
),
|
||||||
_permission(
|
_permission(
|
||||||
ADMIN_SCOPE,
|
ADMIN_SCOPE,
|
||||||
"Administer datasources",
|
"Administer datasources",
|
||||||
@@ -166,6 +177,12 @@ ROLE_TEMPLATES = (
|
|||||||
STAGE_WRITE_SCOPE,
|
STAGE_WRITE_SCOPE,
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
RoleTemplate(
|
||||||
|
slug="datasource_approver",
|
||||||
|
name="Datasource approver",
|
||||||
|
description="Independently approve or reject governed datasource stages.",
|
||||||
|
permissions=(CATALOGUE_READ_SCOPE, STAGE_APPROVE_SCOPE),
|
||||||
|
),
|
||||||
RoleTemplate(
|
RoleTemplate(
|
||||||
slug="datasource_reader",
|
slug="datasource_reader",
|
||||||
name="Datasource reader",
|
name="Datasource reader",
|
||||||
@@ -234,6 +251,14 @@ def _tenant_summary(session, tenant_id: str) -> dict[str, int]:
|
|||||||
)
|
)
|
||||||
.count()
|
.count()
|
||||||
),
|
),
|
||||||
|
"datasource_stages_awaiting_approval": (
|
||||||
|
session.query(datasource_models.DatasourceStageRecord)
|
||||||
|
.filter(
|
||||||
|
datasource_models.DatasourceStageRecord.tenant_id == tenant_id,
|
||||||
|
datasource_models.DatasourceStageRecord.state == "awaiting_approval",
|
||||||
|
)
|
||||||
|
.count()
|
||||||
|
),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -256,6 +281,7 @@ manifest = ModuleManifest(
|
|||||||
CAPABILITY_AUTH_PERMISSION_EVALUATOR,
|
CAPABILITY_AUTH_PERMISSION_EVALUATOR,
|
||||||
CAPABILITY_DATASOURCE_ORIGINS,
|
CAPABILITY_DATASOURCE_ORIGINS,
|
||||||
CAPABILITY_DATASOURCE_ARTIFACT_BACKENDS,
|
CAPABILITY_DATASOURCE_ARTIFACT_BACKENDS,
|
||||||
|
CAPABILITY_POLICY_DATASOURCE_VISIBILITY,
|
||||||
),
|
),
|
||||||
provides_interfaces=(
|
provides_interfaces=(
|
||||||
ModuleInterfaceProvider(
|
ModuleInterfaceProvider(
|
||||||
@@ -293,6 +319,12 @@ manifest = ModuleManifest(
|
|||||||
version_max_exclusive="2.0.0",
|
version_max_exclusive="2.0.0",
|
||||||
optional=True,
|
optional=True,
|
||||||
),
|
),
|
||||||
|
ModuleInterfaceRequirement(
|
||||||
|
name="policy.datasource_visibility",
|
||||||
|
version_min="1.0.0",
|
||||||
|
version_max_exclusive="2.0.0",
|
||||||
|
optional=True,
|
||||||
|
),
|
||||||
),
|
),
|
||||||
permissions=PERMISSIONS,
|
permissions=PERMISSIONS,
|
||||||
role_templates=ROLE_TEMPLATES,
|
role_templates=ROLE_TEMPLATES,
|
||||||
@@ -382,6 +414,20 @@ manifest = ModuleManifest(
|
|||||||
label="Datasource preview and materializations",
|
label="Datasource preview and materializations",
|
||||||
order=50,
|
order=50,
|
||||||
),
|
),
|
||||||
|
ViewSurface(
|
||||||
|
id="datasources.lifecycle-evidence",
|
||||||
|
module_id=MODULE_ID,
|
||||||
|
kind="section",
|
||||||
|
label="Datasource lifecycle evidence",
|
||||||
|
order=60,
|
||||||
|
),
|
||||||
|
ViewSurface(
|
||||||
|
id="datasources.retention",
|
||||||
|
module_id=MODULE_ID,
|
||||||
|
kind="action",
|
||||||
|
label="Datasource retention preview and apply",
|
||||||
|
order=70,
|
||||||
|
),
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
route_factory=_router,
|
route_factory=_router,
|
||||||
@@ -413,6 +459,7 @@ manifest = ModuleManifest(
|
|||||||
script_location=str(Path(__file__).with_name("migrations") / "versions"),
|
script_location=str(Path(__file__).with_name("migrations") / "versions"),
|
||||||
retirement_supported=True,
|
retirement_supported=True,
|
||||||
retirement_provider=drop_table_retirement_provider(
|
retirement_provider=drop_table_retirement_provider(
|
||||||
|
datasource_models.DatasourceLifecycleEvidenceRecord,
|
||||||
datasource_models.DatasourcePublicationRecord,
|
datasource_models.DatasourcePublicationRecord,
|
||||||
datasource_models.DatasourceStageRecord,
|
datasource_models.DatasourceStageRecord,
|
||||||
datasource_models.DatasourceMaterializationRecord,
|
datasource_models.DatasourceMaterializationRecord,
|
||||||
@@ -428,6 +475,7 @@ manifest = ModuleManifest(
|
|||||||
),
|
),
|
||||||
uninstall_guard_providers=(
|
uninstall_guard_providers=(
|
||||||
persistent_table_uninstall_guard(
|
persistent_table_uninstall_guard(
|
||||||
|
datasource_models.DatasourceLifecycleEvidenceRecord,
|
||||||
datasource_models.DatasourceRecord,
|
datasource_models.DatasourceRecord,
|
||||||
datasource_models.DatasourceMaterializationRecord,
|
datasource_models.DatasourceMaterializationRecord,
|
||||||
datasource_models.DatasourcePayloadRecord,
|
datasource_models.DatasourcePayloadRecord,
|
||||||
@@ -439,12 +487,97 @@ manifest = ModuleManifest(
|
|||||||
),
|
),
|
||||||
architecture=ARCHITECTURE,
|
architecture=ARCHITECTURE,
|
||||||
documentation=(
|
documentation=(
|
||||||
|
DocumentationTopic(
|
||||||
|
id="datasources.csv-source-fidelity",
|
||||||
|
title="CSV input fidelity, original export and retention",
|
||||||
|
summary="Keep original upload text alongside durable stages and immutable materializations without bypassing visibility or retention.",
|
||||||
|
body=(
|
||||||
|
"New CSV upload dialogs default to Preserve text; values remain strings, including whitespace, exact decimal digits, large identifiers and explicit empty records. Infer types (legacy) is an explicit compatibility option. "
|
||||||
|
"Existing API calls without csv_value_mode retain their legacy behavior. The API accepts text or legacy_typed; parsed headers are normalized, and malformed text-mode row shapes are rejected. "
|
||||||
|
"Durable CSV stages retain the original supplied UTF-8 text separately from metadata, bind it to the exact scalar types and values of the parsed rows, and include source evidence in approval. Promotion and freezing preserve the original with the corresponding immutable materialization. This does not persist preview rows. "
|
||||||
|
"Original text and rows each have a 5 MB limit; there are at most 10,000 rows. Each stage/materialization, including a frozen copy, can retain up to 5 MB of original text plus serialization and metadata storage overhead. "
|
||||||
|
"GET /api/v1/datasources/{datasource_id}/materializations/{materialization_id}/original-csv is an audited, uncached attachment export for current datasources:source:admin authority. "
|
||||||
|
"Current and historical visibility and policy-provider decisions still apply. Any row filter or denied field prevents raw export, even for an administrator; use governed row previews instead. No raw input is returned in catalogue, stage or history DTOs. "
|
||||||
|
"Freezing verifies the existing original evidence before copying and retains inherited access restrictions; the original may carry at most 32 distinct governance snapshots, with explicit rejection beyond that limit. "
|
||||||
|
"Original text is checksum-verified and follows the same governed retention as its stage/materialization. Stage deletion removes its original; materialization payload disposal also clears that original while retaining audit evidence. "
|
||||||
|
"No original is invented for historical imports. Missing or disposed originals are reported unavailable. Export preserves the API's supplied text as UTF-8, including unsanitized spreadsheet formulas, and is not a reconstruction of an earlier file encoding."
|
||||||
|
),
|
||||||
|
layer="always", documentation_types=("user", "admin"), audience=("user", "module_admin", "operator"), order=8,
|
||||||
|
translations={"de": {
|
||||||
|
"title": "CSV-Original, genauer Import und Aufbewahrung",
|
||||||
|
"summary": "Originaltext mit dauerhaften Staging-Ständen und Materialisierungen erhalten, ohne Sichtbarkeit oder Aufbewahrung zu umgehen.",
|
||||||
|
"body": (
|
||||||
|
"Neue CSV-Importdialoge wählen standardmäßig Text erhalten: Werte einschließlich Leerzeichen, genauer Dezimalstellen, großer Kennungen und ausdrücklich leerer Datensätze bleiben Zeichenketten. Typen ableiten (bisheriges Verhalten) ist eine ausdrückliche Kompatibilitätsoption. "
|
||||||
|
"Bestehende API-Aufrufe ohne csv_value_mode bleiben unverändert; unterstützt werden text und legacy_typed. Überschriften werden normalisiert, fehlerhafte Zeilenformen im Textmodus abgelehnt. "
|
||||||
|
"Dauerhafte CSV-Staging-Stände speichern den gelieferten UTF-8-Originaltext getrennt von Metadaten, prüfen genaue Typen und Werte der abgeleiteten Zeilen und binden den Quellnachweis in die Freigabe ein. Übernahme und Einfrieren erhalten das Original bei der jeweiligen unveränderlichen Materialisierung. Vorschauzeilen werden dadurch nicht gespeichert. "
|
||||||
|
"Original und Zeilen sind jeweils auf 5 MB und die Tabelle auf 10.000 Zeilen begrenzt. Pro Staging-Stand oder Materialisierung einschließlich eingefrorener Kopien werden bis zu 5 MB Originaltext zuzüglich Speicher für Serialisierung und Metadaten aufbewahrt. "
|
||||||
|
"GET /api/v1/datasources/{datasource_id}/materializations/{materialization_id}/original-csv liefert einen protokollierten, nicht zwischengespeicherten Download mit aktuellem Recht datasources:source:admin. "
|
||||||
|
"Aktuelle und historische Sichtbarkeit sowie externe Richtlinienentscheidungen gelten weiterhin. Jeder Zeilenfilter oder ein nicht freigegebenes Feld verhindert den Originalexport auch für die Administration; verwenden Sie dann die gesteuerte Zeilenvorschau. Katalog, Staging- und Verlaufsantworten enthalten keinen Originaltext. "
|
||||||
|
"Einfrieren prüft den vorhandenen Originalnachweis vor dem Kopieren und erhält geerbte Zugriffsbeschränkungen. Höchstens 32 unterschiedliche Richtlinienstände dürfen ein Original begleiten; darüber hinaus wird ausdrücklich abgewiesen. "
|
||||||
|
"Prüfsummen werden vor dem Export geprüft. Das Original folgt der gesteuerten Aufbewahrung: Löschen eines Staging-Stands entfernt dessen Original; Entsorgung einer Materialisierung entfernt auch deren Original, behält aber Auditnachweise. "
|
||||||
|
"Für historische Importe wird kein Original erfunden. Fehlende oder entsorgte Originale werden als nicht verfügbar gemeldet. Der Download erhält den an die API gelieferten Text als UTF-8 einschließlich unveränderter Tabellenformeln, nicht eine frühere Dateikodierung."
|
||||||
|
),
|
||||||
|
}},
|
||||||
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="datasources.detail-freshness",
|
||||||
|
title="Preview and revision history after a datasource change",
|
||||||
|
summary="Reload the selected datasource details even when its catalogue reference stays the same.",
|
||||||
|
body=(
|
||||||
|
"After a successful catalogue reload, the selected datasource preview and materialization history are read again. "
|
||||||
|
"This includes manual Reload and reloads following refresh, freezing, stage promotion or governance changes, even when the selected reference or current materialization is unchanged. "
|
||||||
|
"While reloading, previous rows and history are not presented as the newly selected or updated datasource. "
|
||||||
|
"An older request cannot replace results for a newer selection, reload or authentication context. "
|
||||||
|
"Callbacks retained from an earlier authorization context cannot start a new reload or change success/error messages and selection. "
|
||||||
|
"After a real identity, credential or permission change, the old catalogue is hidden until the new context has loaded it; open dialog inputs are reset to avoid carrying drafts into another authorization context. "
|
||||||
|
"Cosmetic profile changes and ordinary refreshes with equivalent authority do not reset dialogs or trigger duplicate detail reads. "
|
||||||
|
"Preview and history are independent reads: a preview failure can still leave authorized history available. "
|
||||||
|
"Live datasource previews may read the provider through the existing bounded, authorized preview API; this is not background polling or automatic source refresh. "
|
||||||
|
"Rows are not normalized, reordered or persisted by this display update. Snapshot immutability, retention rules and all server-side permission checks remain unchanged."
|
||||||
|
),
|
||||||
|
layer="always", documentation_types=("user", "admin"), audience=("user", "module_admin", "operator"), order=6,
|
||||||
|
translations={"de": {
|
||||||
|
"title": "Vorschau und Revisionsverlauf nach einer Datenquellenänderung",
|
||||||
|
"summary": "Details der ausgewählten Datenquelle auch bei unverändertem Katalogverweis neu laden.",
|
||||||
|
"body": (
|
||||||
|
"Nach erfolgreichem Neuladen des Katalogs werden Vorschau und Materialisierungsverlauf der ausgewählten Datenquelle erneut abgerufen. "
|
||||||
|
"Dies gilt für Neu laden sowie für das Neuladen nach Aktualisierung, Einfrieren, Staging-Übernahme oder Governance-Änderungen, auch wenn Auswahlverweis oder aktuelle Materialisierung gleich bleiben. "
|
||||||
|
"Während des Neuladens werden bisherige Zeilen und Verläufe nicht als neue Auswahl oder aktualisierte Datenquelle dargestellt. "
|
||||||
|
"Eine ältere Anfrage kann Ergebnisse einer neueren Auswahl, eines neueren Abrufs oder eines geänderten Authentifizierungskontexts nicht ersetzen. "
|
||||||
|
"Rückrufe eines früheren Berechtigungskontexts starten keinen neuen Abruf und ändern weder Erfolgs-/Fehlermeldungen noch die Auswahl. "
|
||||||
|
"Nach einem tatsächlichen Identitäts-, Zugangsdaten- oder Rechtewechsel bleibt der alte Katalog verborgen, bis er im neuen Kontext geladen wurde; offene Dialogeingaben werden zurückgesetzt, damit Entwürfe nicht in einen anderen Berechtigungskontext gelangen. "
|
||||||
|
"Rein optische Profiländerungen und gewöhnliche Aktualisierungen mit gleichwertigen Rechten setzen Dialoge nicht zurück und lösen keine doppelten Detailabrufe aus. "
|
||||||
|
"Vorschau und Verlauf sind getrennte Lesezugriffe: Bei einem Vorschaufehler kann ein berechtigter Verlauf weiterhin verfügbar sein. "
|
||||||
|
"Vorschauen lebender Datenquellen können den Anbieter über die bestehende begrenzte und berechtigungsgeprüfte Vorschau-API lesen; dies ist weder Hintergrund-Polling noch eine automatische Quellenaktualisierung. "
|
||||||
|
"Die Anzeigeaktualisierung normalisiert, sortiert oder speichert keine Zeilen dauerhaft. Unveränderlichkeit von Snapshots, Aufbewahrungsregeln und sämtliche serverseitigen Rechteprüfungen bleiben erhalten."
|
||||||
|
),
|
||||||
|
}},
|
||||||
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="datasources.workspace-layout",
|
||||||
|
title="Datasources workspace actions",
|
||||||
|
summary="Find collection-wide commands in their consistent workspace position.",
|
||||||
|
body="Open the documentation book beside Data for workspace guidance; field documentation stays "
|
||||||
|
"beside the relevant label. "
|
||||||
|
"Reload and Add datasource or stage use the persistent full-width workspace header at the upper right; Reload sits immediately before creation. Selecting a record, changing filters, or opening an editor does not move these collection-wide commands into the left pane. Catalogue, staging, and origin filters remain in the collection pane. Snapshot and freeze actions remain with their selected resource. Existing permissions, disabled-state rules, and unsaved-change guards still apply. Administrators configure authority through the existing permission system; no new permission or automatic operation is introduced.",
|
||||||
|
layer="static",
|
||||||
|
documentation_types=("user", "admin"),
|
||||||
|
audience=("user", "module_admin", "operator"),
|
||||||
|
order=5,
|
||||||
|
translations={"de": {
|
||||||
|
"title": "Datenquellen: Aktionen im Arbeitsbereich",
|
||||||
|
"summary": "Sammlungsweite Aktionen an ihrer einheitlichen Position im Arbeitsbereich finden.",
|
||||||
|
"body": "Öffnen Sie das Dokumentationsbuch neben Daten für Hinweise zum Arbeitsbereich; "
|
||||||
|
"Felddokumentation bleibt neben der jeweiligen Feldbezeichnung. "
|
||||||
|
"Neu laden und Datenquelle oder Staging hinzufügen stehen oben rechts in der dauerhaft sichtbaren, arbeitsbereichsweiten Leiste; Neu laden steht unmittelbar vor dem Anlegen. Auswahl, Filterwechsel und Bearbeitung verschieben diese sammlungsweiten Aktionen nicht in den linken Bereich. Katalog-, Staging- und Herkunftsfilter bleiben im Sammlungsbereich. Snapshot- und Einfrieraktionen bleiben bei der ausgewählten Ressource. Bestehende Berechtigungen, Deaktivierungsregeln und der Schutz ungespeicherter Änderungen gelten weiterhin. Administratoren konfigurieren Rechte im bestehenden Berechtigungssystem; es entstehen weder neue Rechte noch automatische Vorgänge.",
|
||||||
|
}},
|
||||||
|
),
|
||||||
DocumentationTopic(
|
DocumentationTopic(
|
||||||
id="datasources.data-subject-requests",
|
id="datasources.data-subject-requests",
|
||||||
title="Datasources data-subject requests",
|
title="Datasources data-subject requests",
|
||||||
summary="Identify governed datasource copies while preserving credential, payload, and immutable-evidence boundaries.",
|
summary="Identify governed datasource copies while preserving credential, payload, and immutable-evidence boundaries.",
|
||||||
body=(
|
body=(
|
||||||
"Datasources matches exact tenant-scoped catalogue, governance-reference, materialization, payload, stage, and publication identifiers plus minimized account, identity, or membership operator attribution. DSAR results never copy connector/provider references, locators, credentials, arbitrary rows, schemas, validation samples, metadata, provenance bodies, checkpoints, idempotency material, or hashes. Arbitrary tabular payloads are not scanned for identifiers because that would be incomplete, schema-dependent, and liable to disclose unrelated people; the authoritative source module must locate and correct subject facts. "
|
"Datasources matches exact tenant-scoped catalogue, governance-reference, materialization, payload, stage, publication, and lifecycle-evidence identifiers plus minimized account, identity, or membership operator attribution. DSAR results never copy connector/provider references, locators, credentials, arbitrary rows, schemas, validation samples, metadata, provenance bodies, checkpoints, idempotency material, or hashes. Arbitrary tabular payloads are not scanned for identifiers because that would be incomplete, schema-dependent, and liable to disclose unrelated people; the authoritative source module must locate and correct subject facts. "
|
||||||
"An unpromoted stage or unreferenced payload can be deleted idempotently. Published catalogue state, promoted stages, referenced payloads, immutable materializations, governance references, publications, holds, and operator attribution require data-steward review and source correction. Downstream Dataflow and Reporting outputs must be refreshed after correction."
|
"An unpromoted stage or unreferenced payload can be deleted idempotently. Published catalogue state, promoted stages, referenced payloads, immutable materializations, governance references, publications, holds, and operator attribution require data-steward review and source correction. Downstream Dataflow and Reporting outputs must be refreshed after correction."
|
||||||
),
|
),
|
||||||
layer="configured",
|
layer="configured",
|
||||||
@@ -452,6 +585,27 @@ manifest = ModuleManifest(
|
|||||||
audience=("user", "operator", "module_admin", "data_steward", "auditor"),
|
audience=("user", "operator", "module_admin", "data_steward", "auditor"),
|
||||||
related_modules=("core", "connectors", "dataflow", "reporting"),
|
related_modules=("core", "connectors", "dataflow", "reporting"),
|
||||||
order=69,
|
order=69,
|
||||||
|
translations={
|
||||||
|
"de": {
|
||||||
|
"title": "Betroffenenanfragen für Datenquellen",
|
||||||
|
"summary": (
|
||||||
|
"Gesteuerte Datenquellenkopien identifizieren und dabei Grenzen für Zugangsdaten, Nutzdaten und unveränderliche "
|
||||||
|
"Nachweise wahren."
|
||||||
|
),
|
||||||
|
"body": (
|
||||||
|
"Datasources gleicht exakte mandantenbezogene Kennungen für Katalog, Governance-Verweis, Materialisierung, Nutzdaten, "
|
||||||
|
"Stufe, Veröffentlichung und Lebenszyklusnachweis sowie minimierte Betriebszuordnungen zu Konto, Identität oder "
|
||||||
|
"Mitgliedschaft ab. DSAR-Ergebnisse kopieren niemals Connector-/Providerverweise, Speicherorte, Zugangsdaten, beliebige "
|
||||||
|
"Zeilen, Schemata, Validierungsstichproben, Metadaten, Herkunftstexte, Checkpoints, Idempotenzmaterial oder Hashes. "
|
||||||
|
"Beliebige tabellarische Nutzdaten werden nicht nach Kennungen durchsucht, weil dies unvollständig und schemaabhängig wäre "
|
||||||
|
"und Daten unbeteiligter Personen offenlegen könnte; das maßgebliche Quellmodul muss Betroffenenmerkmale finden und "
|
||||||
|
"korrigieren. Eine nicht hochgestufte Stufe oder nicht referenzierte Nutzdaten können idempotent gelöscht werden. "
|
||||||
|
"Veröffentlichter Katalogzustand, hochgestufte Stufen, referenzierte Nutzdaten, unveränderliche Materialisierungen, "
|
||||||
|
"Governance-Verweise, Veröffentlichungen, Sperren und Betriebszuordnungen erfordern die Prüfung durch Data Stewards und "
|
||||||
|
"eine Korrektur an der Quelle. Nach der Korrektur müssen nachgelagerte Dataflow- und Reporting-Ausgaben aktualisiert werden."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
},
|
||||||
metadata={
|
metadata={
|
||||||
"seed": True,
|
"seed": True,
|
||||||
"help_contexts": [
|
"help_contexts": [
|
||||||
@@ -497,6 +651,28 @@ manifest = ModuleManifest(
|
|||||||
"search",
|
"search",
|
||||||
),
|
),
|
||||||
order=70,
|
order=70,
|
||||||
|
translations={
|
||||||
|
"de": {
|
||||||
|
"title": "Lebenszyklus einer Datenquelle",
|
||||||
|
"summary": (
|
||||||
|
"Gesteuerte Live-, Cache- und statische Daten-/Registereinträge mit Staging, eingefrorenen Zuständen, Herkunft und "
|
||||||
|
"institutionellem Eigentumskontext."
|
||||||
|
),
|
||||||
|
"body": (
|
||||||
|
"Datasources ist der providerneutrale Katalog für Dataflow, Workflow, Reporting und richtlinienbewusste Module. Statische "
|
||||||
|
"Daten werden vor der Hochstufung bereitgestellt. Cache-Daten aktualisieren Connector-Ursprünge in unveränderliche "
|
||||||
|
"Materialisierungen. Live-Daten werden über einen Connector gelesen und können für reproduzierbare Nachweise eingefroren "
|
||||||
|
"werden. Connectors besitzt Protokolle und Zugangsdaten; Datasources besitzt Datenidentität, Herkunft, Lebenszyklus, "
|
||||||
|
"Lesesemantik und den typisierten Governance-Katalog für Hoheit, Zweck, Qualität, Aktualität, Klassifikation, Korrektur "
|
||||||
|
"sowie abhängige Services, Flows, Berichte, Kontrollen und Entscheidungen. Für Live-Vorschauen bewahrt es Quellmodus, "
|
||||||
|
"strukturierten Gesundheitszustand, deklarierte Pushdown-Unterstützung und wirksame Zeilen-, Byte- und Zeitgrenzen. "
|
||||||
|
"Sichtbarkeit von Governance-Metadaten gewährt keinen Zugriff auf geschützte Zeilen. Ist Search aktiviert, indiziert das "
|
||||||
|
"Modul nur begrenzte Katalogbezeichnungen und governance-sichere Facetten und prüft vor einem Treffer die aktuelle "
|
||||||
|
"Katalogberechtigung erneut. Zeilen, Schemata, Connector-Verweise, Zugangsdaten, beliebige Metadaten und Herkunft werden "
|
||||||
|
"niemals in den Suchindex kopiert."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
},
|
||||||
metadata={
|
metadata={
|
||||||
"seed": True,
|
"seed": True,
|
||||||
"help_contexts": [
|
"help_contexts": [
|
||||||
@@ -516,8 +692,8 @@ manifest = ModuleManifest(
|
|||||||
"Authority mode states whether GovOPlaN, an external system, a synchronized projection, an overlay, or a linked reference "
|
"Authority mode states whether GovOPlaN, an external system, a synchronized projection, an overlay, or a linked reference "
|
||||||
"controls the data. The authoritative source, owner, steward, responsible organization/function, schema owner, privacy "
|
"controls the data. The authoritative source, owner, steward, responsible organization/function, schema owner, privacy "
|
||||||
"profile, retention policy, transfer agreement, legal basis, holds, correction procedure, purposes, official keys, and "
|
"profile, retention policy, transfer agreement, legal basis, holds, correction procedure, purposes, official keys, and "
|
||||||
"known limits provide discoverable institutional context. A retention-policy reference identifies the owning Policy rule; it "
|
"known limits provide discoverable institutional context. A retention-policy reference identifies an owning external Policy rule, while "
|
||||||
"does not itself delete materializations, override legal holds, or prove that a scheduler is active. A transfer-agreement "
|
"the local versioned retention contract controls previewable stage and payload disposition. Retention never overrides legal holds, the current materialization, or publication evidence. A transfer-agreement "
|
||||||
"reference records the governed agreement for external exchange; it does not grant connector credentials, recipient access, "
|
"reference records the governed agreement for external exchange; it does not grant connector credentials, recipient access, "
|
||||||
"or authority to export classified rows. Freshness and quality policies are typed JSON contracts retained "
|
"or authority to export classified rows. Freshness and quality policies are typed JSON contracts retained "
|
||||||
"with materialization evidence. Datasources enforces the declared bounded tabular stage rules and schema policy; origin-specific "
|
"with materialization evidence. Datasources enforces the declared bounded tabular stage rules and schema policy; origin-specific "
|
||||||
@@ -536,6 +712,30 @@ manifest = ModuleManifest(
|
|||||||
"risk_compliance",
|
"risk_compliance",
|
||||||
),
|
),
|
||||||
order=71,
|
order=71,
|
||||||
|
translations={
|
||||||
|
"de": {
|
||||||
|
"title": "Datenhoheit und Governance einer Datenquelle",
|
||||||
|
"summary": (
|
||||||
|
"Erläutern, wer Datenbedeutung, Hoheit, Korrektur, Datenschutz, Qualität, Aktualität, Aufbewahrung und abhängige "
|
||||||
|
"Verwendungen verantwortet."
|
||||||
|
),
|
||||||
|
"body": (
|
||||||
|
"Der Hoheitsmodus gibt an, ob GovOPlaN, ein externes System, eine synchronisierte Projektion, ein Overlay oder ein "
|
||||||
|
"verknüpfter Verweis die Daten steuert. Maßgebliche Quelle, Eigentum, Data Steward, verantwortliche Organisation/Funktion, "
|
||||||
|
"Schemaeigentum, Datenschutzprofil, Aufbewahrungsrichtlinie, Übermittlungsvereinbarung, Rechtsgrundlage, Sperren, "
|
||||||
|
"Korrekturverfahren, Zwecke, amtliche Schlüssel und bekannte Grenzen bilden den auffindbaren institutionellen Kontext. "
|
||||||
|
"Ein Aufbewahrungsrichtlinienverweis bezeichnet eine externe Regel im Eigentum von Policy; der lokale versionierte "
|
||||||
|
"Aufbewahrungsvertrag steuert die vorschaufähige Behandlung von Stufen und Nutzdaten. Aufbewahrung überstimmt niemals "
|
||||||
|
"gesetzliche Sperren, die aktuelle Materialisierung oder Veröffentlichungsnachweise. Ein Verweis auf eine "
|
||||||
|
"Übermittlungsvereinbarung dokumentiert die gesteuerte Vereinbarung für externen Austausch; er gewährt weder "
|
||||||
|
"Connector-Zugangsdaten noch Empfängerzugriff oder Befugnis zum Export klassifizierter Zeilen. Aktualitäts- und "
|
||||||
|
"Qualitätsrichtlinien sind typisierte JSON-Verträge, die mit Materialisierungsnachweisen aufbewahrt werden. Datasources "
|
||||||
|
"setzt die deklarierten begrenzten Regeln für tabellarische Stufen und die Schemarichtlinie durch; ursprungs- oder "
|
||||||
|
"verbraucherspezifische Kontrollen verbleiben beim Provider oder der nutzenden Kontrolle, die Unterstützung erklärt. "
|
||||||
|
"Metadatensichtbarkeit gewährt niemals Zeilenzugriff."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
},
|
||||||
metadata={
|
metadata={
|
||||||
"seed": True,
|
"seed": True,
|
||||||
"help_contexts": [
|
"help_contexts": [
|
||||||
@@ -547,10 +747,64 @@ manifest = ModuleManifest(
|
|||||||
"datasources.field.freshness-policy",
|
"datasources.field.freshness-policy",
|
||||||
"datasources.field.quality-policy",
|
"datasources.field.quality-policy",
|
||||||
"datasources.field.retention-policy",
|
"datasources.field.retention-policy",
|
||||||
|
"datasources.field.access-policy",
|
||||||
|
"datasources.field.visibility-policy",
|
||||||
"datasources.field.transfer-agreement",
|
"datasources.field.transfer-agreement",
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
),
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="datasources.visibility",
|
||||||
|
title="Datasource and field visibility",
|
||||||
|
summary="Apply source, materialization, field, and row controls before protected rows leave Datasources.",
|
||||||
|
body=(
|
||||||
|
"A local visibility policy can restrict discovery and reads by account, membership, identity, group, role, service account, or authentication method. "
|
||||||
|
"Materialization ACLs additionally protect current and frozen revisions. Field rules classify a field and either replace its value with null or omit the field unless an allow ACL matches. Row filters compare a configured field with a principal account, membership, identity, service-account, group, role, or function-assignment claim; all configured filters and all policy overlays must allow a row. "
|
||||||
|
"Datasources applies these controls before rows leave its provider boundary and returns an opaque permitted-view fingerprint bound to the base revision, effective policies, and relevant principal facts. Denied and filtered reads emit audit evidence containing outcomes and policy diagnostics, never protected values, provider locators, arbitrary metadata, or provenance bodies. "
|
||||||
|
"An optional access-policy reference delegates hierarchical overlays to Policy. Policy can only tighten local behavior. An unresolved reference, unavailable configured provider, or malformed policy fails closed. Without Policy and without a reference, the local visibility policy remains fully usable. Frozen reads enforce the current local policy together with a distinct restrictive policy captured in the frozen governance snapshot."
|
||||||
|
),
|
||||||
|
layer="available",
|
||||||
|
documentation_types=("admin", "user"),
|
||||||
|
audience=("operator", "module_admin", "data_steward", "auditor"),
|
||||||
|
related_modules=("policy", "access", "audit", "connectors"),
|
||||||
|
order=72,
|
||||||
|
translations={
|
||||||
|
"de": {
|
||||||
|
"title": "Sichtbarkeit von Datenquellen und Feldern",
|
||||||
|
"summary": (
|
||||||
|
"Kontrollen für Quelle, Materialisierung, Feld und Zeile anwenden, bevor geschützte Zeilen Datasources verlassen."
|
||||||
|
),
|
||||||
|
"body": (
|
||||||
|
"Eine lokale Sichtbarkeitsrichtlinie kann Auffinden und Lesen nach Konto, Mitgliedschaft, Identität, Gruppe, Rolle, "
|
||||||
|
"Dienstkonto oder Authentifizierungsmethode einschränken. Materialisierungs-ACLs schützen zusätzlich aktuelle und "
|
||||||
|
"eingefrorene Revisionen. Feldregeln klassifizieren ein Feld und ersetzen seinen Wert durch null oder lassen es weg, "
|
||||||
|
"sofern keine passende Erlaubnis-ACL vorliegt. Zeilenfilter vergleichen ein konfiguriertes Feld mit Konto-, "
|
||||||
|
"Mitgliedschafts-, Identitäts-, Dienstkonto-, Gruppen-, Rollen- oder Funktionszuweisungsmerkmalen der Hauptperson; alle "
|
||||||
|
"konfigurierten Filter und Richtlinienebenen müssen eine Zeile erlauben. Datasources wendet diese Kontrollen an, bevor "
|
||||||
|
"Zeilen die Providergrenze verlassen, und liefert einen undurchsichtigen Fingerabdruck der erlaubten Ansicht, gebunden an "
|
||||||
|
"Basisrevision, wirksame Richtlinien und relevante Hauptpersonenmerkmale. Verweigerte und gefilterte Lesevorgänge erzeugen "
|
||||||
|
"Auditnachweise mit Ergebnissen und Richtliniendiagnosen, niemals geschützten Werten, Provider-Speicherorten, beliebigen "
|
||||||
|
"Metadaten oder Herkunftstexten. Ein optionaler Zugriffsrichtlinienverweis delegiert hierarchische Ebenen an Policy, das "
|
||||||
|
"lokales Verhalten nur verschärfen kann. Eine nicht auflösbare Referenz, ein nicht verfügbarer konfigurierter Provider oder "
|
||||||
|
"eine fehlerhafte Richtlinie schließt sicher. Ohne Policy und ohne Referenz bleibt die lokale Richtlinie vollständig "
|
||||||
|
"nutzbar. Eingefrorene Lesevorgänge setzen die aktuelle lokale Richtlinie gemeinsam mit einer getrennten einschränkenden "
|
||||||
|
"Richtlinie aus dem eingefrorenen Governance-Snapshot durch."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
},
|
||||||
|
metadata={
|
||||||
|
"seed": True,
|
||||||
|
"help_contexts": [
|
||||||
|
"datasources.field.access-policy",
|
||||||
|
"datasources.field.visibility-policy",
|
||||||
|
"datasources.preview",
|
||||||
|
],
|
||||||
|
"limitations": [
|
||||||
|
"Row filters support deterministic equality or membership against principal claims; arbitrary expressions are not executed.",
|
||||||
|
"Live filtered previews scan at most 10,000 origin rows and disclose a structured limitation diagnostic if the origin is larger.",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
),
|
||||||
DocumentationTopic(
|
DocumentationTopic(
|
||||||
id="datasources.quality-gates",
|
id="datasources.quality-gates",
|
||||||
title="Validate staged and produced Datasource revisions",
|
title="Validate staged and produced Datasource revisions",
|
||||||
@@ -565,8 +819,7 @@ manifest = ModuleManifest(
|
|||||||
"durable provider-neutral artifact reference with a pinned locator, SHA-256 checksum, declared schema, fingerprint, and size. "
|
"durable provider-neutral artifact reference with a pinned locator, SHA-256 checksum, declared schema, fingerprint, and size. "
|
||||||
"The configured payload backend verifies the artifact and provides bounded reads. Content-level rules require checksum- and "
|
"The configured payload backend verifies the artifact and provides bounded reads. Content-level rules require checksum- and "
|
||||||
"policy-bound producer evidence; absent evidence creates an immutable review-required materialization without changing the "
|
"policy-bound producer evidence; absent evidence creates an immutable review-required materialization without changing the "
|
||||||
"Datasource's current state. Warning and review-required outcomes are preserved for Workflow handoffs. Approval and retention execution "
|
"Datasource's current state. Warning and review-required outcomes are preserved for Workflow handoffs. Versioned approval policy can require an attributable, separated quorum before a stage or refresh becomes current; retention uses a fresh, hashed preview before any explicit administrator apply."
|
||||||
"are not inferred from arbitrary JSON flags and remain separate governed lifecycle work."
|
|
||||||
),
|
),
|
||||||
layer="available",
|
layer="available",
|
||||||
documentation_types=("admin", "user"),
|
documentation_types=("admin", "user"),
|
||||||
@@ -585,7 +838,32 @@ manifest = ModuleManifest(
|
|||||||
"dataflow",
|
"dataflow",
|
||||||
"workflow_engine",
|
"workflow_engine",
|
||||||
),
|
),
|
||||||
order=72,
|
order=73,
|
||||||
|
translations={
|
||||||
|
"de": {
|
||||||
|
"title": "Bereitgestellte und erzeugte Datenquellenrevisionen validieren",
|
||||||
|
"summary": (
|
||||||
|
"Deterministische Qualitätsregeln und Schemerichtlinien anwenden, bevor bereitgestellte oder erzeugte Daten nutzbar werden."
|
||||||
|
),
|
||||||
|
"body": (
|
||||||
|
"Tabellarische Stufen prüfen konfigurierte Zeilenanzahl, Pflichtfelder, Feldform, Nullbarkeit, Eindeutigkeit, Zahlenbereiche "
|
||||||
|
"und begrenzte Referenzmengen. Fehler halten die Stufe prüfbar, blockieren aber die Hochstufung; Warnungen bleiben sichtbar "
|
||||||
|
"und erlauben eine ausdrückliche Hochstufung. Aktualisierungen vergleichen das erkannte Schema mit dem aktuellen Ziel und "
|
||||||
|
"klassifizieren jede Änderung als kompatibel, Warnung oder brechend. Diagnosen zeigen Anzahlen und begrenzte Zeilennummern, "
|
||||||
|
"niemals Feldwerte. Richtlinienversion und -hash, Diagnosen und Schemadifferenz werden bei erfolgreicher Hochstufung in die "
|
||||||
|
"unveränderliche Materialisierungsherkunft kopiert. Eine Provider-Veröffentlichung wendet dasselbe Gate vor jeder "
|
||||||
|
"Katalogwirkung an, bewahrt Validierungsnachweise an der unveränderlichen Ausgaberevision, serialisiert die "
|
||||||
|
"Veröffentlichungsidentität über PostgreSQL-Worker und sendet ein transaktionales Abschlussereignis. Große Ausgaben können "
|
||||||
|
"stattdessen einen dauerhaften providerneutralen Artefaktverweis mit festgelegtem Speicherort, SHA-256-Prüfsumme, Schema, "
|
||||||
|
"Fingerabdruck und Größe liefern. Das konfigurierte Nutzdaten-Backend prüft das Artefakt und stellt begrenzte Lesevorgänge "
|
||||||
|
"bereit. Inhaltsregeln verlangen prüfsummen- und richtliniengebundene Provider-Nachweise; fehlen sie, entsteht eine "
|
||||||
|
"unveränderliche prüfpflichtige Materialisierung, ohne den aktuellen Zustand der Datenquelle zu verändern. Warnungen und "
|
||||||
|
"Prüfpflicht bleiben für Workflow-Übergaben erhalten. Eine versionierte Freigaberichtlinie kann ein zurechenbares, "
|
||||||
|
"getrenntes Quorum verlangen, bevor eine Stufe oder Aktualisierung aktuell wird; Aufbewahrung verwendet vor jeder "
|
||||||
|
"ausdrücklichen administrativen Anwendung eine frische gehashte Vorschau."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
},
|
||||||
metadata={
|
metadata={
|
||||||
"seed": True,
|
"seed": True,
|
||||||
"help_contexts": [
|
"help_contexts": [
|
||||||
@@ -596,7 +874,8 @@ manifest = ModuleManifest(
|
|||||||
],
|
],
|
||||||
"limitations": [
|
"limitations": [
|
||||||
"Referential rules currently use a bounded embedded value set rather than reading another protected Datasource.",
|
"Referential rules currently use a bounded embedded value set rather than reading another protected Datasource.",
|
||||||
"Approval authority and automatic retention execution are not part of the current stage contract.",
|
"Approvals are module-local lifecycle evidence; installing a separate Approvals module does not silently change configured datasource policy.",
|
||||||
|
"Retention is never automatic: an administrator must preview a current plan and explicitly apply eligible targets.",
|
||||||
"Artifact bytes remain owned by their payload backend; Datasources stores an immutable reference and integrity evidence.",
|
"Artifact bytes remain owned by their payload backend; Datasources stores an immutable reference and integrity evidence.",
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
@@ -609,7 +888,7 @@ manifest = ModuleManifest(
|
|||||||
"A Datasource key is the stable catalogue identity used by consumers. Live mode reads through an available origin; cached "
|
"A Datasource key is the stable catalogue identity used by consumers. Live mode reads through an available origin; cached "
|
||||||
"mode refreshes an origin into immutable revisions; static mode promotes uploaded content from staging. Stages are bounded, "
|
"mode refreshes an origin into immutable revisions; static mode promotes uploaded content from staging. Stages are bounded, "
|
||||||
"inspectable, and non-consumable until promoted. Promotion creates or updates a governed Datasource and appends an immutable "
|
"inspectable, and non-consumable until promoted. Promotion creates or updates a governed Datasource and appends an immutable "
|
||||||
"materialization. Refresh appends a new cached revision without rewriting older evidence. Freeze labels an immutable, "
|
"materialization. When approval is configured, a refresh first creates a non-consumable stage and only an approved quorum permits promotion. Freeze labels an immutable, "
|
||||||
"addressable state for reproducible execution. Retirement removes the Datasource from new definitions while retained "
|
"addressable state for reproducible execution. Retirement removes the Datasource from new definitions while retained "
|
||||||
"materialization references remain governed. Connector absence disables origin registration but leaves local catalogue and "
|
"materialization references remain governed. Connector absence disables origin registration but leaves local catalogue and "
|
||||||
"staging behavior available."
|
"staging behavior available."
|
||||||
@@ -624,8 +903,25 @@ manifest = ModuleManifest(
|
|||||||
"reporting",
|
"reporting",
|
||||||
"audit",
|
"audit",
|
||||||
),
|
),
|
||||||
order=73,
|
order=74,
|
||||||
|
translations={
|
||||||
|
"de": {
|
||||||
|
"title": "Datenquellenfelder und Folgen des Lebenszyklus",
|
||||||
|
"summary": "Semantik von Live-, Cache- und statischem Modus, Staging, Hochstufung, Aktualisierung, Einfrieren und Stilllegung.",
|
||||||
|
"body": (
|
||||||
|
"Der Schlüssel einer Datenquelle ist ihre stabile Katalogidentität für Verbraucher. Live-Modus liest über einen "
|
||||||
|
"verfügbaren Ursprung, Cache-Modus aktualisiert einen Ursprung in unveränderliche Revisionen und statischer Modus stuft "
|
||||||
|
"hochgeladene Inhalte aus dem Staging hoch. Stufen sind begrenzt, prüfbar und vor der Hochstufung nicht nutzbar. Die "
|
||||||
|
"Hochstufung legt eine gesteuerte Datenquelle an oder aktualisiert sie und ergänzt eine unveränderliche Materialisierung. "
|
||||||
|
"Ist eine Freigabe konfiguriert, erzeugt eine Aktualisierung zunächst eine nicht nutzbare Stufe; nur ein bestätigtes Quorum "
|
||||||
|
"erlaubt die Hochstufung. Einfrieren kennzeichnet einen unveränderlichen adressierbaren Zustand für reproduzierbare "
|
||||||
|
"Ausführung. Eine Stilllegung entfernt die Datenquelle aus neuen Definitionen, während erhaltene Materialisierungsverweise "
|
||||||
|
"gesteuert bleiben. Fehlt Connectors, ist die Ursprungsregistrierung deaktiviert; lokaler Katalog und Staging bleiben nutzbar."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
},
|
||||||
metadata={
|
metadata={
|
||||||
|
"kind": "reference",
|
||||||
"seed": True,
|
"seed": True,
|
||||||
"help_contexts": [
|
"help_contexts": [
|
||||||
"datasources.field.origin",
|
"datasources.field.origin",
|
||||||
@@ -644,7 +940,62 @@ manifest = ModuleManifest(
|
|||||||
},
|
},
|
||||||
},
|
},
|
||||||
),
|
),
|
||||||
|
DocumentationTopic(
|
||||||
|
id="datasources.approval-and-retention",
|
||||||
|
title="Approve promotions and apply retention safely",
|
||||||
|
summary="Require separated datasource decisions and preview retention consequences before disposing staged or materialized payloads.",
|
||||||
|
body=(
|
||||||
|
"A versioned approval policy can require one to five distinct approvals before a valid stage becomes ready. The stage creator cannot approve when separation of duties is enabled. Decisions are bound to the exact stage fingerprint, quality-policy hash, approval-policy hash, actor authority, reason, and expiry; a changed policy or staged subject invalidates promotion. Cached refreshes use the same stage path whenever approval is required. "
|
||||||
|
"A versioned retention policy independently sets durations for stages, ordinary materializations, and frozen evidence. Preview returns a hashed plan with every due target and blocker. Current materializations, legal holds, pending approvals, and producer-publication evidence cannot be disposed. Applying the unchanged plan deletes eligible transient stages or purges materialization payload rows while retaining minimized schema/provenance, checksum, disposition, and hash-chained lifecycle evidence. Empty local policies disable both gates, so the module remains usable without Access or Policy; installed modules may tighten access but do not manufacture approval claims."
|
||||||
),
|
),
|
||||||
|
layer="available",
|
||||||
|
documentation_types=("admin", "user"),
|
||||||
|
audience=("operator", "module_admin", "data_steward", "auditor"),
|
||||||
|
conditions=(
|
||||||
|
DocumentationCondition(
|
||||||
|
any_scopes=(
|
||||||
|
STAGE_WRITE_SCOPE,
|
||||||
|
STAGE_APPROVE_SCOPE,
|
||||||
|
ADMIN_SCOPE,
|
||||||
|
)
|
||||||
|
),
|
||||||
|
),
|
||||||
|
related_modules=("access", "audit", "policy", "workflow_engine"),
|
||||||
|
order=75,
|
||||||
|
translations={
|
||||||
|
"de": {
|
||||||
|
"title": "Freigaben erteilen und Aufbewahrung sicher anwenden",
|
||||||
|
"summary": "Getrennte Entscheidungen für Datenquellen verlangen und Folgen der Aufbewahrung vor dem Löschen von Stufen oder Nutzdaten prüfen.",
|
||||||
|
"body": (
|
||||||
|
"Eine versionierte Freigaberichtlinie kann ein bis fünf verschiedene Freigaben verlangen, bevor eine gültige Stufe bereit ist. Bei aktivierter Funktionstrennung darf die erstellende Person nicht selbst freigeben. Entscheidungen sind an Fingerabdruck, Qualitäts- und Freigaberichtlinien-Hash, Berechtigung, Begründung und Ablaufzeit gebunden; geänderte Richtlinien oder Inhalte verhindern die Übernahme. Zwischengespeicherte Aktualisierungen nutzen bei Freigabepflicht denselben Stufenweg. "
|
||||||
|
"Eine getrennte Aufbewahrungsrichtlinie legt Fristen für Stufen, gewöhnliche Materialisierungen und eingefrorene Nachweise fest. Die Vorschau liefert einen gehashten Plan mit fälligen Zielen und Sperrgründen. Aktuelle Materialisierungen, rechtliche Sperren, offene Freigaben und Veröffentlichungsnachweise dürfen nicht entfernt werden. Beim Anwenden des unveränderten Plans werden nur zulässige Stufen oder Nutzdaten gelöscht; minimierte Metadaten, Prüfsummen, Dispositionsangaben und hashverkettete Lebenszyklusnachweise bleiben erhalten. Leere lokale Richtlinien deaktivieren beide Schranken, sodass das Modul ohne Access oder Policy nutzbar bleibt."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
},
|
||||||
|
metadata={
|
||||||
|
"seed": True,
|
||||||
|
"kind": "workflow",
|
||||||
|
"help_contexts": [
|
||||||
|
"datasources.staging.approval",
|
||||||
|
"datasources.action.approve",
|
||||||
|
"datasources.action.reject",
|
||||||
|
"datasources.field.approval-policy",
|
||||||
|
"datasources.field.retention-policy-contract",
|
||||||
|
"datasources.lifecycle-evidence",
|
||||||
|
"datasources.retention",
|
||||||
|
],
|
||||||
|
"limitations": [
|
||||||
|
"Retention execution is explicit and plan-bound; this module does not run a hidden deletion scheduler.",
|
||||||
|
"Disposed materialization records retain minimized schema and provenance metadata while their payload rows are removed.",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
manifest = with_documentation_structured_translations(
|
||||||
|
manifest, locale="de", translations=GERMAN_STRUCTURED_TRANSLATIONS
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+38
@@ -0,0 +1,38 @@
|
|||||||
|
"""v0.1.20 datasource visibility policy
|
||||||
|
|
||||||
|
Revision ID: c9e3a6f1d4b8
|
||||||
|
Revises: b8d2f5a0c3e7
|
||||||
|
Create Date: 2026-08-21 20:15:00.000000
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision = "c9e3a6f1d4b8"
|
||||||
|
down_revision = "b8d2f5a0c3e7"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.add_column(
|
||||||
|
"datasource_catalogue",
|
||||||
|
sa.Column("access_policy_ref", sa.String(length=500), nullable=True),
|
||||||
|
)
|
||||||
|
op.add_column(
|
||||||
|
"datasource_catalogue",
|
||||||
|
sa.Column(
|
||||||
|
"visibility_policy",
|
||||||
|
sa.JSON(),
|
||||||
|
server_default=sa.text("'{}'"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_column("datasource_catalogue", "visibility_policy")
|
||||||
|
op.drop_column("datasource_catalogue", "access_policy_ref")
|
||||||
+142
@@ -0,0 +1,142 @@
|
|||||||
|
"""v0.1.21 datasource lifecycle governance
|
||||||
|
|
||||||
|
Revision ID: d1a7c3e9f5b2
|
||||||
|
Revises: c9e3a6f1d4b8
|
||||||
|
Create Date: 2026-08-22 20:00:00.000000
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
revision = "d1a7c3e9f5b2"
|
||||||
|
down_revision = "c9e3a6f1d4b8"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.add_column(
|
||||||
|
"datasource_catalogue",
|
||||||
|
sa.Column(
|
||||||
|
"approval_policy",
|
||||||
|
sa.JSON(),
|
||||||
|
server_default=sa.text("'{}'"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.add_column(
|
||||||
|
"datasource_catalogue",
|
||||||
|
sa.Column(
|
||||||
|
"retention_policy",
|
||||||
|
sa.JSON(),
|
||||||
|
server_default=sa.text("'{}'"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.add_column(
|
||||||
|
"datasource_stages",
|
||||||
|
sa.Column(
|
||||||
|
"approval",
|
||||||
|
sa.JSON(),
|
||||||
|
server_default=sa.text("'{}'"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.add_column(
|
||||||
|
"datasource_materializations",
|
||||||
|
sa.Column("disposed_at", sa.DateTime(timezone=True), nullable=True),
|
||||||
|
)
|
||||||
|
op.add_column(
|
||||||
|
"datasource_materializations",
|
||||||
|
sa.Column(
|
||||||
|
"disposition",
|
||||||
|
sa.JSON(),
|
||||||
|
server_default=sa.text("'{}'"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_materializations_disposed_at",
|
||||||
|
"datasource_materializations",
|
||||||
|
["disposed_at"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_table(
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
sa.Column("id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||||
|
sa.Column("subject_ref", sa.String(length=160), nullable=False),
|
||||||
|
sa.Column("event_type", sa.String(length=80), nullable=False),
|
||||||
|
sa.Column("occurred_at", sa.DateTime(timezone=True), nullable=False),
|
||||||
|
sa.Column("actor_ref", sa.String(length=255), nullable=True),
|
||||||
|
sa.Column("policy_version", sa.String(length=120), nullable=True),
|
||||||
|
sa.Column("policy_hash", sa.String(length=64), nullable=True),
|
||||||
|
sa.Column("subject_digest", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column("previous_event_hash", sa.String(length=64), nullable=True),
|
||||||
|
sa.Column("event_hash", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column("details", sa.JSON(), nullable=False),
|
||||||
|
sa.PrimaryKeyConstraint("id"),
|
||||||
|
sa.UniqueConstraint(
|
||||||
|
"tenant_id",
|
||||||
|
"event_hash",
|
||||||
|
name="uq_datasource_lifecycle_evidence_hash",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_lifecycle_evidence_tenant_id",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
["tenant_id"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_lifecycle_evidence_subject_ref",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
["subject_ref"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_lifecycle_evidence_event_type",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
["event_type"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_lifecycle_evidence_actor_ref",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
["actor_ref"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_lifecycle_evidence_event_hash",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
["event_hash"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_lifecycle_evidence_subject",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
["tenant_id", "subject_ref", "occurred_at"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
"ix_datasource_lifecycle_evidence_event",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
["tenant_id", "event_type", "occurred_at"],
|
||||||
|
unique=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_table("datasource_lifecycle_evidence")
|
||||||
|
op.drop_index(
|
||||||
|
"ix_datasource_materializations_disposed_at",
|
||||||
|
table_name="datasource_materializations",
|
||||||
|
)
|
||||||
|
op.drop_column("datasource_materializations", "disposition")
|
||||||
|
op.drop_column("datasource_materializations", "disposed_at")
|
||||||
|
op.drop_column("datasource_stages", "approval")
|
||||||
|
op.drop_column("datasource_catalogue", "retention_policy")
|
||||||
|
op.drop_column("datasource_catalogue", "approval_policy")
|
||||||
+24
@@ -0,0 +1,24 @@
|
|||||||
|
"""Preserve original CSV for new durable imports, separate from catalogue DTOs."""
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
revision = "e2b8d4a0f6c3"
|
||||||
|
down_revision = "d1a7c3e9f5b2"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.add_column(
|
||||||
|
"datasource_stages", sa.Column("csv_source", sa.JSON(), nullable=True)
|
||||||
|
)
|
||||||
|
op.add_column(
|
||||||
|
"datasource_materializations", sa.Column("csv_source", sa.JSON(), nullable=True)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
for table in ("datasource_materializations", "datasource_stages"):
|
||||||
|
with op.batch_alter_table(table) as batch:
|
||||||
|
batch.drop_column("csv_source")
|
||||||
@@ -1,8 +1,11 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
from collections.abc import Mapping
|
from collections.abc import Mapping
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
from fastapi import APIRouter, Depends, HTTPException, Query, Response, status
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from govoplan_core.audit.logging import audit_event
|
from govoplan_core.audit.logging import audit_event
|
||||||
@@ -22,12 +25,15 @@ from govoplan_core.core.datasources import (
|
|||||||
datasource_origins,
|
datasource_origins,
|
||||||
)
|
)
|
||||||
from govoplan_core.db.session import get_session
|
from govoplan_core.db.session import get_session
|
||||||
|
from govoplan_core.core.tabular_sources import TabularCsvSource
|
||||||
from govoplan_datasources.backend.runtime import get_registry
|
from govoplan_datasources.backend.runtime import get_registry
|
||||||
from govoplan_datasources.backend.schemas import (
|
from govoplan_datasources.backend.schemas import (
|
||||||
DatasourceFieldResponse,
|
DatasourceFieldResponse,
|
||||||
DatasourceFreezeRequest,
|
DatasourceFreezeRequest,
|
||||||
DatasourceGovernancePayload,
|
DatasourceGovernancePayload,
|
||||||
DatasourceGovernanceUpdateRequest,
|
DatasourceGovernanceUpdateRequest,
|
||||||
|
DatasourceLifecycleEvidenceListResponse,
|
||||||
|
DatasourceLifecycleEvidenceResponse,
|
||||||
DatasourceListResponse,
|
DatasourceListResponse,
|
||||||
DatasourceMaterializationListResponse,
|
DatasourceMaterializationListResponse,
|
||||||
DatasourceMaterializationResponse,
|
DatasourceMaterializationResponse,
|
||||||
@@ -39,8 +45,13 @@ from govoplan_datasources.backend.schemas import (
|
|||||||
DatasourcePreviewDiagnosticResponse,
|
DatasourcePreviewDiagnosticResponse,
|
||||||
DatasourcePreviewResponse,
|
DatasourcePreviewResponse,
|
||||||
DatasourceResponse,
|
DatasourceResponse,
|
||||||
|
DatasourceRetentionApplyRequest,
|
||||||
|
DatasourceRetentionApplyResponse,
|
||||||
|
DatasourceRetentionCandidateResponse,
|
||||||
|
DatasourceRetentionPlanResponse,
|
||||||
DatasourceRetireResponse,
|
DatasourceRetireResponse,
|
||||||
DatasourceStageCreateRequest,
|
DatasourceStageCreateRequest,
|
||||||
|
DatasourceStageDecisionRequest,
|
||||||
DatasourceStageListResponse,
|
DatasourceStageListResponse,
|
||||||
DatasourceStagePromoteRequest,
|
DatasourceStagePromoteRequest,
|
||||||
DatasourceStagePromoteResponse,
|
DatasourceStagePromoteResponse,
|
||||||
@@ -50,6 +61,7 @@ from govoplan_datasources.backend.service import (
|
|||||||
ADMIN_SCOPE,
|
ADMIN_SCOPE,
|
||||||
CATALOGUE_READ_SCOPE,
|
CATALOGUE_READ_SCOPE,
|
||||||
SOURCE_WRITE_SCOPE,
|
SOURCE_WRITE_SCOPE,
|
||||||
|
STAGE_APPROVE_SCOPE,
|
||||||
STAGE_WRITE_SCOPE,
|
STAGE_WRITE_SCOPE,
|
||||||
SqlDatasourceProvider,
|
SqlDatasourceProvider,
|
||||||
)
|
)
|
||||||
@@ -192,7 +204,7 @@ def api_create_stage(
|
|||||||
rows = (
|
rows = (
|
||||||
tuple(payload.rows or ())
|
tuple(payload.rows or ())
|
||||||
if payload.format == "json"
|
if payload.format == "json"
|
||||||
else parse_csv_rows(payload.csv_text or "", delimiter=payload.delimiter)
|
else parse_csv_rows(payload.csv_text or "", delimiter=payload.delimiter, value_mode=payload.csv_value_mode)
|
||||||
)
|
)
|
||||||
stage = _provider().create_stage(
|
stage = _provider().create_stage(
|
||||||
session,
|
session,
|
||||||
@@ -214,6 +226,12 @@ def api_create_stage(
|
|||||||
},
|
},
|
||||||
metadata=payload.metadata,
|
metadata=payload.metadata,
|
||||||
governance=_governance(payload.governance),
|
governance=_governance(payload.governance),
|
||||||
|
csv_source=(TabularCsvSource(
|
||||||
|
text=payload.csv_text or "",
|
||||||
|
delimiter=payload.delimiter,
|
||||||
|
value_mode=payload.csv_value_mode,
|
||||||
|
parser_profile="datasources.csv.v1",
|
||||||
|
) if payload.format == "csv" else None),
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
except DatasourceError as exc:
|
except DatasourceError as exc:
|
||||||
@@ -234,6 +252,56 @@ def api_create_stage(
|
|||||||
"schema_classification": _safe_mapping(
|
"schema_classification": _safe_mapping(
|
||||||
stage.validation.get("schema_change")
|
stage.validation.get("schema_change")
|
||||||
).get("classification"),
|
).get("classification"),
|
||||||
|
"approval_state": stage.approval.get("state"),
|
||||||
|
"approval_policy_hash": stage.approval.get("policy_hash"),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return _stage_response(stage)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/stages/{stage_id}/decision",
|
||||||
|
response_model=DatasourceStageResponse,
|
||||||
|
)
|
||||||
|
def api_decide_stage(
|
||||||
|
stage_id: str,
|
||||||
|
payload: DatasourceStageDecisionRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> DatasourceStageResponse:
|
||||||
|
_require_any_scope(principal, STAGE_APPROVE_SCOPE, ADMIN_SCOPE)
|
||||||
|
try:
|
||||||
|
stage, evidence_hash, replayed = _provider().decide_stage(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
stage_ref=f"stage:{stage_id}",
|
||||||
|
decision=payload.decision,
|
||||||
|
reason=payload.reason,
|
||||||
|
expected_policy_hash=payload.expected_policy_hash,
|
||||||
|
expected_subject_digest=payload.expected_subject_digest,
|
||||||
|
)
|
||||||
|
except DatasourceError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
action = (
|
||||||
|
"datasources.stage.approved"
|
||||||
|
if payload.decision == "approve"
|
||||||
|
else "datasources.stage.rejected"
|
||||||
|
)
|
||||||
|
_audit(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action=action,
|
||||||
|
object_type="datasource_stage",
|
||||||
|
object_id=stage.ref,
|
||||||
|
details={
|
||||||
|
"decision": payload.decision,
|
||||||
|
"resulting_state": stage.state,
|
||||||
|
"approval_count": stage.approval.get("approval_count"),
|
||||||
|
"required_approvals": stage.approval.get("required_approvals"),
|
||||||
|
"policy_hash": stage.approval.get("policy_hash"),
|
||||||
|
"evidence_hash": evidence_hash,
|
||||||
|
"replayed": replayed,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
session.commit()
|
session.commit()
|
||||||
@@ -275,6 +343,12 @@ def api_promote_stage(
|
|||||||
"quality_policy_hash": _safe_mapping(
|
"quality_policy_hash": _safe_mapping(
|
||||||
materialization.provenance.get("stage_validation")
|
materialization.provenance.get("stage_validation")
|
||||||
).get("policy_hash"),
|
).get("policy_hash"),
|
||||||
|
"approval_policy_hash": _safe_mapping(
|
||||||
|
materialization.provenance.get("stage_approval")
|
||||||
|
).get("policy_hash"),
|
||||||
|
"promotion_evidence_hash": materialization.provenance.get(
|
||||||
|
"promotion_evidence_hash"
|
||||||
|
),
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
session.commit()
|
session.commit()
|
||||||
@@ -284,6 +358,100 @@ def api_promote_stage(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/lifecycle-evidence",
|
||||||
|
response_model=DatasourceLifecycleEvidenceListResponse,
|
||||||
|
)
|
||||||
|
def api_list_lifecycle_evidence(
|
||||||
|
subject_ref: str | None = Query(default=None, max_length=160),
|
||||||
|
limit: int = Query(default=200, ge=1, le=500),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> DatasourceLifecycleEvidenceListResponse:
|
||||||
|
_require_any_scope(principal, CATALOGUE_READ_SCOPE, ADMIN_SCOPE)
|
||||||
|
try:
|
||||||
|
rows = _provider().list_lifecycle_evidence(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
subject_ref=subject_ref,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
except DatasourceError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
return DatasourceLifecycleEvidenceListResponse(
|
||||||
|
evidence=[_evidence_response(item) for item in rows]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/retention/plan",
|
||||||
|
response_model=DatasourceRetentionPlanResponse,
|
||||||
|
)
|
||||||
|
def api_preview_retention(
|
||||||
|
as_of: datetime | None = Query(default=None),
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> DatasourceRetentionPlanResponse:
|
||||||
|
_require_any_scope(principal, ADMIN_SCOPE)
|
||||||
|
effective_at = as_of or datetime.now(UTC)
|
||||||
|
try:
|
||||||
|
plan = _provider().preview_retention(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
as_of=effective_at,
|
||||||
|
)
|
||||||
|
except DatasourceError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
return DatasourceRetentionPlanResponse(
|
||||||
|
as_of=plan.as_of.isoformat(),
|
||||||
|
plan_hash=plan.plan_hash,
|
||||||
|
candidates=[
|
||||||
|
DatasourceRetentionCandidateResponse(**item.to_dict())
|
||||||
|
for item in plan.candidates
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/retention/apply",
|
||||||
|
response_model=DatasourceRetentionApplyResponse,
|
||||||
|
)
|
||||||
|
def api_apply_retention(
|
||||||
|
payload: DatasourceRetentionApplyRequest,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> DatasourceRetentionApplyResponse:
|
||||||
|
_require_any_scope(principal, ADMIN_SCOPE)
|
||||||
|
try:
|
||||||
|
disposed, evidence_hashes = _provider().apply_retention(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
as_of=payload.as_of,
|
||||||
|
plan_hash=payload.plan_hash,
|
||||||
|
target_refs=payload.target_refs,
|
||||||
|
)
|
||||||
|
except DatasourceError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
_audit(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="datasources.retention.applied",
|
||||||
|
object_type="datasource_retention_plan",
|
||||||
|
object_id=payload.plan_hash,
|
||||||
|
details={
|
||||||
|
"as_of": payload.as_of.isoformat(),
|
||||||
|
"disposed_refs": list(disposed),
|
||||||
|
"evidence_hashes": list(evidence_hashes),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return DatasourceRetentionApplyResponse(
|
||||||
|
plan_hash=payload.plan_hash,
|
||||||
|
disposed_refs=list(disposed),
|
||||||
|
evidence_hashes=list(evidence_hashes),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.get("", response_model=DatasourceListResponse)
|
@router.get("", response_model=DatasourceListResponse)
|
||||||
def api_list_datasources(
|
def api_list_datasources(
|
||||||
query: str = Query(default="", max_length=200),
|
query: str = Query(default="", max_length=200),
|
||||||
@@ -370,7 +538,17 @@ def api_update_datasource_governance(
|
|||||||
action="datasources.governance.updated",
|
action="datasources.governance.updated",
|
||||||
object_type="datasource",
|
object_type="datasource",
|
||||||
object_id=item.ref,
|
object_id=item.ref,
|
||||||
details={"governance": item.governance.to_dict()},
|
details={
|
||||||
|
"classification": item.governance.classification,
|
||||||
|
"publication_state": item.governance.publication_state,
|
||||||
|
"access_policy_ref": item.governance.access_policy_ref,
|
||||||
|
"visibility_policy_configured": bool(
|
||||||
|
item.governance.visibility_policy
|
||||||
|
),
|
||||||
|
"visibility_policy_hash": _mapping_hash(
|
||||||
|
item.governance.visibility_policy
|
||||||
|
),
|
||||||
|
},
|
||||||
)
|
)
|
||||||
session.commit()
|
session.commit()
|
||||||
return _datasource_response(item)
|
return _datasource_response(item)
|
||||||
@@ -458,6 +636,27 @@ def api_list_materializations(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/{datasource_id}/materializations/{materialization_id}/original-csv")
|
||||||
|
def api_original_csv(
|
||||||
|
datasource_id: str,
|
||||||
|
materialization_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> Response:
|
||||||
|
_require_any_scope(principal, ADMIN_SCOPE)
|
||||||
|
try:
|
||||||
|
text = _provider().original_csv(
|
||||||
|
session, principal,
|
||||||
|
datasource_ref=f"datasource:{datasource_id}",
|
||||||
|
materialization_ref=f"materialization:{materialization_id}",
|
||||||
|
)
|
||||||
|
except DatasourceError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
_audit(session, principal, action="datasources.original_csv.exported", object_type="datasource_materialization", object_id=f"materialization:{materialization_id}", details={"datasource_ref": f"datasource:{datasource_id}", "sha256": hashlib.sha256(text.encode("utf-8")).hexdigest()})
|
||||||
|
session.commit()
|
||||||
|
return Response(content=text.encode("utf-8"), media_type="text/csv; charset=utf-8", headers={"Content-Disposition": 'attachment; filename="original.csv"', "Cache-Control": "no-store"})
|
||||||
|
|
||||||
|
|
||||||
@router.post(
|
@router.post(
|
||||||
"/{datasource_id}/refresh",
|
"/{datasource_id}/refresh",
|
||||||
response_model=DatasourceStagePromoteResponse,
|
response_model=DatasourceStagePromoteResponse,
|
||||||
@@ -495,6 +694,46 @@ def api_refresh_datasource(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/{datasource_id}/refresh/stage",
|
||||||
|
response_model=DatasourceStageResponse,
|
||||||
|
status_code=status.HTTP_201_CREATED,
|
||||||
|
)
|
||||||
|
def api_prepare_refresh(
|
||||||
|
datasource_id: str,
|
||||||
|
session: Session = Depends(get_session),
|
||||||
|
principal: ApiPrincipal = Depends(get_api_principal),
|
||||||
|
) -> DatasourceStageResponse:
|
||||||
|
_require_any_scope(
|
||||||
|
principal,
|
||||||
|
SOURCE_WRITE_SCOPE,
|
||||||
|
STAGE_WRITE_SCOPE,
|
||||||
|
ADMIN_SCOPE,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
stage = _provider().prepare_refresh(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
datasource_ref=f"datasource:{datasource_id}",
|
||||||
|
)
|
||||||
|
except DatasourceError as exc:
|
||||||
|
raise _http_error(exc) from exc
|
||||||
|
_audit(
|
||||||
|
session,
|
||||||
|
principal,
|
||||||
|
action="datasources.refresh.staged",
|
||||||
|
object_type="datasource_stage",
|
||||||
|
object_id=stage.ref,
|
||||||
|
details={
|
||||||
|
"datasource_ref": stage.target_datasource_ref,
|
||||||
|
"approval_state": stage.approval.get("state"),
|
||||||
|
"quality_policy_hash": stage.validation.get("policy_hash"),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
return _stage_response(stage)
|
||||||
|
|
||||||
|
|
||||||
@router.post(
|
@router.post(
|
||||||
"/{datasource_id}/freeze",
|
"/{datasource_id}/freeze",
|
||||||
response_model=DatasourceMaterializationResponse,
|
response_model=DatasourceMaterializationResponse,
|
||||||
@@ -580,6 +819,7 @@ def _datasource_response(item: DatasourceDescriptor) -> DatasourceResponse:
|
|||||||
name=field.name,
|
name=field.name,
|
||||||
data_type=field.data_type,
|
data_type=field.data_type,
|
||||||
nullable=field.nullable,
|
nullable=field.nullable,
|
||||||
|
classification=field.classification,
|
||||||
)
|
)
|
||||||
for field in item.schema
|
for field in item.schema
|
||||||
],
|
],
|
||||||
@@ -610,6 +850,7 @@ def _materialization_response(
|
|||||||
name=field.name,
|
name=field.name,
|
||||||
data_type=field.data_type,
|
data_type=field.data_type,
|
||||||
nullable=field.nullable,
|
nullable=field.nullable,
|
||||||
|
classification=field.classification,
|
||||||
)
|
)
|
||||||
for field in item.schema
|
for field in item.schema
|
||||||
],
|
],
|
||||||
@@ -621,6 +862,8 @@ def _materialization_response(
|
|||||||
item.source_timestamp.isoformat() if item.source_timestamp else None
|
item.source_timestamp.isoformat() if item.source_timestamp else None
|
||||||
),
|
),
|
||||||
created_at=item.created_at.isoformat() if item.created_at else None,
|
created_at=item.created_at.isoformat() if item.created_at else None,
|
||||||
|
disposed_at=item.disposed_at.isoformat() if item.disposed_at else None,
|
||||||
|
disposition=dict(item.disposition),
|
||||||
provenance=dict(item.provenance),
|
provenance=dict(item.provenance),
|
||||||
metadata=dict(item.metadata),
|
metadata=dict(item.metadata),
|
||||||
governance=item.governance.to_dict(),
|
governance=item.governance.to_dict(),
|
||||||
@@ -643,12 +886,14 @@ def _stage_response(item: DatasourceStage) -> DatasourceStageResponse:
|
|||||||
name=field.name,
|
name=field.name,
|
||||||
data_type=field.data_type,
|
data_type=field.data_type,
|
||||||
nullable=field.nullable,
|
nullable=field.nullable,
|
||||||
|
classification=field.classification,
|
||||||
)
|
)
|
||||||
for field in item.schema
|
for field in item.schema
|
||||||
],
|
],
|
||||||
row_count=item.row_count,
|
row_count=item.row_count,
|
||||||
byte_count=item.byte_count,
|
byte_count=item.byte_count,
|
||||||
validation=dict(item.validation),
|
validation=dict(item.validation),
|
||||||
|
approval=dict(item.approval),
|
||||||
created_at=item.created_at.isoformat() if item.created_at else None,
|
created_at=item.created_at.isoformat() if item.created_at else None,
|
||||||
promoted_at=item.promoted_at.isoformat() if item.promoted_at else None,
|
promoted_at=item.promoted_at.isoformat() if item.promoted_at else None,
|
||||||
promoted_materialization_ref=item.promoted_materialization_ref,
|
promoted_materialization_ref=item.promoted_materialization_ref,
|
||||||
@@ -673,6 +918,7 @@ def _origin_response(item: DatasourceOrigin) -> DatasourceOriginResponse:
|
|||||||
name=field.name,
|
name=field.name,
|
||||||
data_type=field.data_type,
|
data_type=field.data_type,
|
||||||
nullable=field.nullable,
|
nullable=field.nullable,
|
||||||
|
classification=field.classification,
|
||||||
)
|
)
|
||||||
for field in item.schema
|
for field in item.schema
|
||||||
],
|
],
|
||||||
@@ -703,6 +949,22 @@ def _origin_response(item: DatasourceOrigin) -> DatasourceOriginResponse:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _evidence_response(item) -> DatasourceLifecycleEvidenceResponse:
|
||||||
|
return DatasourceLifecycleEvidenceResponse(
|
||||||
|
ref=f"lifecycle-evidence:{item.id}",
|
||||||
|
subject_ref=item.subject_ref,
|
||||||
|
event_type=item.event_type,
|
||||||
|
occurred_at=item.occurred_at.isoformat(),
|
||||||
|
actor_ref=item.actor_ref,
|
||||||
|
policy_version=item.policy_version,
|
||||||
|
policy_hash=item.policy_hash,
|
||||||
|
subject_digest=item.subject_digest,
|
||||||
|
previous_event_hash=item.previous_event_hash,
|
||||||
|
event_hash=item.event_hash,
|
||||||
|
details=dict(item.details_),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _governance(
|
def _governance(
|
||||||
payload: DatasourceGovernancePayload | None,
|
payload: DatasourceGovernancePayload | None,
|
||||||
) -> DatasourceGovernance | None:
|
) -> DatasourceGovernance | None:
|
||||||
@@ -715,6 +977,13 @@ def _safe_mapping(value: object) -> Mapping[str, object]:
|
|||||||
return value if isinstance(value, Mapping) else {}
|
return value if isinstance(value, Mapping) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def _mapping_hash(value: Mapping[str, object]) -> str | None:
|
||||||
|
if not value:
|
||||||
|
return None
|
||||||
|
encoded = json.dumps(value, sort_keys=True, separators=(",", ":"), default=str)
|
||||||
|
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
def _audit(
|
def _audit(
|
||||||
session: Session,
|
session: Session,
|
||||||
principal: ApiPrincipal,
|
principal: ApiPrincipal,
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
from typing import Any, Literal
|
from typing import Any, Literal
|
||||||
|
|
||||||
from pydantic import BaseModel, Field, model_validator
|
from pydantic import BaseModel, Field, model_validator
|
||||||
@@ -43,11 +44,15 @@ class DatasourceGovernancePayload(BaseModel):
|
|||||||
classification: str = Field(default="internal", min_length=1, max_length=80)
|
classification: str = Field(default="internal", min_length=1, max_length=80)
|
||||||
privacy_profile_ref: str | None = Field(default=None, max_length=500)
|
privacy_profile_ref: str | None = Field(default=None, max_length=500)
|
||||||
retention_policy_ref: str | None = Field(default=None, max_length=500)
|
retention_policy_ref: str | None = Field(default=None, max_length=500)
|
||||||
|
access_policy_ref: str | None = Field(default=None, max_length=500)
|
||||||
|
visibility_policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
hold_refs: list[str] = Field(default_factory=list, max_length=100)
|
hold_refs: list[str] = Field(default_factory=list, max_length=100)
|
||||||
publication_state: str = Field(default="draft", min_length=1, max_length=50)
|
publication_state: str = Field(default="draft", min_length=1, max_length=50)
|
||||||
transfer_agreement_ref: str | None = Field(default=None, max_length=500)
|
transfer_agreement_ref: str | None = Field(default=None, max_length=500)
|
||||||
freshness_policy: dict[str, Any] = Field(default_factory=dict)
|
freshness_policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
quality_policy: dict[str, Any] = Field(default_factory=dict)
|
quality_policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
approval_policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
retention_policy: dict[str, Any] = Field(default_factory=dict)
|
||||||
known_limits: list[str] = Field(default_factory=list, max_length=100)
|
known_limits: list[str] = Field(default_factory=list, max_length=100)
|
||||||
correction_procedure_ref: str | None = Field(default=None, max_length=500)
|
correction_procedure_ref: str | None = Field(default=None, max_length=500)
|
||||||
affected_refs: list[str] = Field(default_factory=list, max_length=250)
|
affected_refs: list[str] = Field(default_factory=list, max_length=250)
|
||||||
@@ -70,6 +75,7 @@ class DatasourceFieldResponse(BaseModel):
|
|||||||
name: str
|
name: str
|
||||||
data_type: str
|
data_type: str
|
||||||
nullable: bool
|
nullable: bool
|
||||||
|
classification: str = "internal"
|
||||||
|
|
||||||
|
|
||||||
class DatasourceResponse(BaseModel):
|
class DatasourceResponse(BaseModel):
|
||||||
@@ -113,6 +119,8 @@ class DatasourceMaterializationResponse(BaseModel):
|
|||||||
frozen_label: str | None
|
frozen_label: str | None
|
||||||
source_timestamp: str | None
|
source_timestamp: str | None
|
||||||
created_at: str | None
|
created_at: str | None
|
||||||
|
disposed_at: str | None
|
||||||
|
disposition: dict[str, Any]
|
||||||
provenance: dict[str, Any]
|
provenance: dict[str, Any]
|
||||||
metadata: dict[str, Any]
|
metadata: dict[str, Any]
|
||||||
governance: DatasourceGovernancePayload
|
governance: DatasourceGovernancePayload
|
||||||
@@ -178,6 +186,7 @@ class DatasourceStageResponse(BaseModel):
|
|||||||
row_count: int | None
|
row_count: int | None
|
||||||
byte_count: int | None
|
byte_count: int | None
|
||||||
validation: DatasourceStageValidationResponse
|
validation: DatasourceStageValidationResponse
|
||||||
|
approval: dict[str, Any]
|
||||||
created_at: str | None
|
created_at: str | None
|
||||||
promoted_at: str | None
|
promoted_at: str | None
|
||||||
promoted_materialization_ref: str | None
|
promoted_materialization_ref: str | None
|
||||||
@@ -200,6 +209,7 @@ class DatasourceStageCreateRequest(BaseModel):
|
|||||||
format: Literal["json", "csv"] = "json"
|
format: Literal["json", "csv"] = "json"
|
||||||
rows: list[dict[str, Any]] | None = Field(default=None, max_length=10_000)
|
rows: list[dict[str, Any]] | None = Field(default=None, max_length=10_000)
|
||||||
csv_text: str | None = Field(default=None, max_length=5_000_000)
|
csv_text: str | None = Field(default=None, max_length=5_000_000)
|
||||||
|
csv_value_mode: Literal["legacy_typed", "text"] = "legacy_typed"
|
||||||
delimiter: Literal[",", ";", "\t", "|"] = ","
|
delimiter: Literal[",", ";", "\t", "|"] = ","
|
||||||
target_datasource_ref: str | None = None
|
target_datasource_ref: str | None = None
|
||||||
provenance: dict[str, Any] = Field(default_factory=dict)
|
provenance: dict[str, Any] = Field(default_factory=dict)
|
||||||
@@ -220,6 +230,61 @@ class DatasourceStagePromoteRequest(BaseModel):
|
|||||||
frozen_label: str | None = Field(default=None, max_length=300)
|
frozen_label: str | None = Field(default=None, max_length=300)
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceStageDecisionRequest(BaseModel):
|
||||||
|
decision: Literal["approve", "reject"]
|
||||||
|
reason: str = Field(min_length=1, max_length=2_000)
|
||||||
|
expected_policy_hash: str = Field(min_length=64, max_length=64)
|
||||||
|
expected_subject_digest: str = Field(min_length=64, max_length=64)
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceLifecycleEvidenceResponse(BaseModel):
|
||||||
|
ref: str
|
||||||
|
subject_ref: str
|
||||||
|
event_type: str
|
||||||
|
occurred_at: str
|
||||||
|
actor_ref: str | None
|
||||||
|
policy_version: str | None
|
||||||
|
policy_hash: str | None
|
||||||
|
subject_digest: str
|
||||||
|
previous_event_hash: str | None
|
||||||
|
event_hash: str
|
||||||
|
details: dict[str, Any]
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceLifecycleEvidenceListResponse(BaseModel):
|
||||||
|
evidence: list[DatasourceLifecycleEvidenceResponse]
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceRetentionCandidateResponse(BaseModel):
|
||||||
|
ref: str
|
||||||
|
kind: Literal["stage", "materialization"]
|
||||||
|
datasource_ref: str | None
|
||||||
|
disposition: Literal["delete_stage", "purge_materialization_payload"]
|
||||||
|
eligible_at: str
|
||||||
|
eligible: bool
|
||||||
|
blockers: list[str]
|
||||||
|
policy_version: str
|
||||||
|
policy_hash: str
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceRetentionPlanResponse(BaseModel):
|
||||||
|
as_of: str
|
||||||
|
plan_hash: str
|
||||||
|
candidates: list[DatasourceRetentionCandidateResponse]
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceRetentionApplyRequest(BaseModel):
|
||||||
|
as_of: datetime
|
||||||
|
plan_hash: str = Field(min_length=64, max_length=64)
|
||||||
|
target_refs: list[str] = Field(min_length=1, max_length=500)
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceRetentionApplyResponse(BaseModel):
|
||||||
|
plan_hash: str
|
||||||
|
disposed_refs: list[str]
|
||||||
|
evidence_hashes: list[str]
|
||||||
|
|
||||||
|
|
||||||
class DatasourceStagePromoteResponse(BaseModel):
|
class DatasourceStagePromoteResponse(BaseModel):
|
||||||
datasource: DatasourceResponse
|
datasource: DatasourceResponse
|
||||||
materialization: DatasourceMaterializationResponse
|
materialization: DatasourceMaterializationResponse
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -10,6 +10,8 @@ from decimal import Decimal
|
|||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from govoplan_core.core.datasources import DatasourceField, DatasourceValidationError
|
from govoplan_core.core.datasources import DatasourceField, DatasourceValidationError
|
||||||
|
from govoplan_core.core.tabular_sources import CsvValueMode, TabularSourceError, parse_tabular_csv
|
||||||
|
from govoplan_core.core.tabular_sources import infer_tabular_schema, tabular_type_name as _type_name
|
||||||
|
|
||||||
|
|
||||||
MAX_STAGE_ROWS = 10_000
|
MAX_STAGE_ROWS = 10_000
|
||||||
@@ -36,10 +38,22 @@ def parse_csv_rows(
|
|||||||
csv_text: str,
|
csv_text: str,
|
||||||
*,
|
*,
|
||||||
delimiter: str = ",",
|
delimiter: str = ",",
|
||||||
|
value_mode: CsvValueMode = "legacy_typed",
|
||||||
) -> tuple[dict[str, Any], ...]:
|
) -> tuple[dict[str, Any], ...]:
|
||||||
if delimiter not in {",", ";", "\t", "|"}:
|
if delimiter not in {",", ";", "\t", "|"}:
|
||||||
raise DatasourceValidationError("Unsupported CSV delimiter.")
|
raise DatasourceValidationError("Unsupported CSV delimiter.")
|
||||||
if len(csv_text.encode("utf-8")) > MAX_STAGE_BYTES:
|
if value_mode == "text":
|
||||||
|
try:
|
||||||
|
return normalize_rows(parse_tabular_csv(csv_text, delimiter=delimiter, value_mode="text", max_rows=MAX_STAGE_ROWS, max_bytes=MAX_STAGE_BYTES))
|
||||||
|
except TabularSourceError as exc:
|
||||||
|
raise DatasourceValidationError(str(exc)) from exc
|
||||||
|
if value_mode != "legacy_typed":
|
||||||
|
raise DatasourceValidationError("Unsupported CSV value mode.")
|
||||||
|
try:
|
||||||
|
encoded_text = csv_text.encode("utf-8")
|
||||||
|
except UnicodeError as exc:
|
||||||
|
raise DatasourceValidationError("CSV input must be valid Unicode encodable as UTF-8.") from exc
|
||||||
|
if len(encoded_text) > MAX_STAGE_BYTES:
|
||||||
raise DatasourceValidationError(
|
raise DatasourceValidationError(
|
||||||
f"Staging is limited to {MAX_STAGE_BYTES // 1_000_000} MB."
|
f"Staging is limited to {MAX_STAGE_BYTES // 1_000_000} MB."
|
||||||
)
|
)
|
||||||
@@ -58,6 +72,8 @@ def parse_csv_rows(
|
|||||||
raise DatasourceValidationError("CSV headers must be unique.")
|
raise DatasourceValidationError("CSV headers must be unique.")
|
||||||
rows: list[dict[str, object]] = []
|
rows: list[dict[str, object]] = []
|
||||||
for line_number, values in enumerate(reader, start=2):
|
for line_number, values in enumerate(reader, start=2):
|
||||||
|
if len(rows) >= MAX_STAGE_ROWS:
|
||||||
|
raise DatasourceValidationError(f"Staging is limited to {MAX_STAGE_ROWS:,} rows.")
|
||||||
if len(values) != len(header):
|
if len(values) != len(header):
|
||||||
raise DatasourceValidationError(
|
raise DatasourceValidationError(
|
||||||
f"CSV row {line_number} has {len(values)} values; expected {len(header)}."
|
f"CSV row {line_number} has {len(values)} values; expected {len(header)}."
|
||||||
@@ -76,26 +92,7 @@ def parse_csv_rows(
|
|||||||
def infer_schema(
|
def infer_schema(
|
||||||
rows: Sequence[Mapping[str, object]],
|
rows: Sequence[Mapping[str, object]],
|
||||||
) -> tuple[DatasourceField, ...]:
|
) -> tuple[DatasourceField, ...]:
|
||||||
names: list[str] = []
|
return tuple(DatasourceField(name=column.name, data_type=column.data_type, nullable=column.nullable) for column in infer_tabular_schema(rows, type_name=_type_name))
|
||||||
for row in rows:
|
|
||||||
for name in row:
|
|
||||||
if name not in names:
|
|
||||||
names.append(name)
|
|
||||||
result: list[DatasourceField] = []
|
|
||||||
for name in names:
|
|
||||||
values = [row.get(name) for row in rows]
|
|
||||||
concrete = [value for value in values if value is not None]
|
|
||||||
data_type = _type_name(concrete[0]) if concrete else "unknown"
|
|
||||||
if any(_type_name(value) != data_type for value in concrete[1:]):
|
|
||||||
data_type = "mixed"
|
|
||||||
result.append(
|
|
||||||
DatasourceField(
|
|
||||||
name=name,
|
|
||||||
data_type=data_type,
|
|
||||||
nullable=len(concrete) != len(values),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return tuple(result)
|
|
||||||
|
|
||||||
|
|
||||||
def fingerprint_rows(
|
def fingerprint_rows(
|
||||||
@@ -145,6 +142,7 @@ def field_payload(field: DatasourceField) -> dict[str, object]:
|
|||||||
"name": field.name,
|
"name": field.name,
|
||||||
"data_type": field.data_type,
|
"data_type": field.data_type,
|
||||||
"nullable": field.nullable,
|
"nullable": field.nullable,
|
||||||
|
"classification": field.classification,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -167,22 +165,6 @@ def _json_value(value: object) -> object:
|
|||||||
raise TypeError(f"{type(value).__name__} is not JSON serializable")
|
raise TypeError(f"{type(value).__name__} is not JSON serializable")
|
||||||
|
|
||||||
|
|
||||||
def _type_name(value: object) -> str:
|
|
||||||
if isinstance(value, bool):
|
|
||||||
return "boolean"
|
|
||||||
if isinstance(value, int):
|
|
||||||
return "integer"
|
|
||||||
if isinstance(value, (float, Decimal)):
|
|
||||||
return "number"
|
|
||||||
if isinstance(value, str):
|
|
||||||
return "string"
|
|
||||||
if isinstance(value, list):
|
|
||||||
return "array"
|
|
||||||
if isinstance(value, dict):
|
|
||||||
return "object"
|
|
||||||
return type(value).__name__.lower()
|
|
||||||
|
|
||||||
|
|
||||||
def _csv_value(value: str) -> object:
|
def _csv_value(value: str) -> object:
|
||||||
cleaned = value.strip()
|
cleaned = value.strip()
|
||||||
if not cleaned:
|
if not cleaned:
|
||||||
|
|||||||
@@ -0,0 +1,490 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import cast
|
||||||
|
|
||||||
|
from govoplan_core.core.access import PrincipalRef
|
||||||
|
from govoplan_core.core.datasources import (
|
||||||
|
DatasourceAccessError,
|
||||||
|
DatasourceField,
|
||||||
|
DatasourceValidationError,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.tabular_sources import TabularPreviewDiagnostic
|
||||||
|
|
||||||
|
|
||||||
|
_ACL_KEYS = frozenset(
|
||||||
|
{
|
||||||
|
"account_ids",
|
||||||
|
"membership_ids",
|
||||||
|
"identity_ids",
|
||||||
|
"group_ids",
|
||||||
|
"role_ids",
|
||||||
|
"service_account_ids",
|
||||||
|
"auth_methods",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
_CLAIM_KEYS = frozenset(
|
||||||
|
{
|
||||||
|
"account_id",
|
||||||
|
"membership_id",
|
||||||
|
"identity_id",
|
||||||
|
"service_account_id",
|
||||||
|
"group_ids",
|
||||||
|
"role_ids",
|
||||||
|
"function_assignment_ids",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
_POLICY_KEYS = frozenset({"source_acl", "materialization_acl", "fields", "row_filters"})
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class FieldRule:
|
||||||
|
name: str
|
||||||
|
classification: str
|
||||||
|
action: str
|
||||||
|
allowed: bool
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class RowFilter:
|
||||||
|
field: str
|
||||||
|
claim: str
|
||||||
|
operator: str
|
||||||
|
allow_null: bool
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class VisibilityPlan:
|
||||||
|
applied: bool
|
||||||
|
policy_hashes: tuple[str, ...]
|
||||||
|
field_rules: tuple[FieldRule, ...]
|
||||||
|
row_filters: tuple[RowFilter, ...]
|
||||||
|
principal_fingerprint: str
|
||||||
|
decision_refs: tuple[str, ...] = ()
|
||||||
|
|
||||||
|
def view_fingerprint(self, base_fingerprint: str) -> str:
|
||||||
|
if not self.applied:
|
||||||
|
return base_fingerprint
|
||||||
|
return _digest(
|
||||||
|
{
|
||||||
|
"base_fingerprint": base_fingerprint,
|
||||||
|
"policy_hashes": self.policy_hashes,
|
||||||
|
"principal_fingerprint": self.principal_fingerprint,
|
||||||
|
"decision_refs": self.decision_refs,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def visibility_plan(
|
||||||
|
*,
|
||||||
|
principal: PrincipalRef,
|
||||||
|
policies: Sequence[Mapping[str, object]],
|
||||||
|
materialized: bool,
|
||||||
|
schema: Sequence[DatasourceField],
|
||||||
|
decision_refs: Sequence[str] = (),
|
||||||
|
) -> VisibilityPlan:
|
||||||
|
normalized = tuple(_normalize_policy(policy) for policy in policies if policy)
|
||||||
|
if not normalized:
|
||||||
|
return VisibilityPlan(
|
||||||
|
applied=False,
|
||||||
|
policy_hashes=(),
|
||||||
|
field_rules=(),
|
||||||
|
row_filters=(),
|
||||||
|
principal_fingerprint="",
|
||||||
|
)
|
||||||
|
|
||||||
|
schema_names = {field.name for field in schema}
|
||||||
|
field_rules: list[FieldRule] = []
|
||||||
|
row_filters: list[RowFilter] = []
|
||||||
|
for policy in normalized:
|
||||||
|
source_acl = policy.get("source_acl")
|
||||||
|
if isinstance(source_acl, Mapping) and not _acl_allows(source_acl, principal):
|
||||||
|
raise DatasourceAccessError("Datasource visibility policy denied access.")
|
||||||
|
materialization_acl = policy.get("materialization_acl")
|
||||||
|
if (
|
||||||
|
materialized
|
||||||
|
and isinstance(materialization_acl, Mapping)
|
||||||
|
and not _acl_allows(materialization_acl, principal)
|
||||||
|
):
|
||||||
|
raise DatasourceAccessError(
|
||||||
|
"Datasource materialization policy denied access."
|
||||||
|
)
|
||||||
|
fields = policy.get("fields", {})
|
||||||
|
if isinstance(fields, Mapping):
|
||||||
|
for name, raw_rule in fields.items():
|
||||||
|
if name not in schema_names:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Visibility policy references unknown field {name!r}."
|
||||||
|
)
|
||||||
|
assert isinstance(raw_rule, Mapping)
|
||||||
|
field_rules.append(
|
||||||
|
FieldRule(
|
||||||
|
name=name,
|
||||||
|
classification=str(raw_rule["classification"]),
|
||||||
|
action=str(raw_rule["action"]),
|
||||||
|
allowed=_acl_allows(raw_rule["allow"], principal),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
for raw_filter in policy.get("row_filters", ()):
|
||||||
|
assert isinstance(raw_filter, Mapping)
|
||||||
|
field = str(raw_filter["field"])
|
||||||
|
if field not in schema_names:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Visibility policy references unknown row-filter field {field!r}."
|
||||||
|
)
|
||||||
|
row_filters.append(
|
||||||
|
RowFilter(
|
||||||
|
field=field,
|
||||||
|
claim=str(raw_filter["claim"]),
|
||||||
|
operator=str(raw_filter["operator"]),
|
||||||
|
allow_null=bool(raw_filter["allow_null"]),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
all_facts = _principal_policy_facts(principal)
|
||||||
|
relevant_fact_keys = _relevant_policy_fact_keys(normalized)
|
||||||
|
principal_payload = {key: all_facts[key] for key in sorted(relevant_fact_keys)}
|
||||||
|
return VisibilityPlan(
|
||||||
|
applied=True,
|
||||||
|
policy_hashes=tuple(_digest(policy) for policy in normalized),
|
||||||
|
field_rules=tuple(field_rules),
|
||||||
|
row_filters=tuple(row_filters),
|
||||||
|
principal_fingerprint=_digest(principal_payload),
|
||||||
|
decision_refs=tuple(sorted(set(decision_refs))),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_visibility_policy(
|
||||||
|
policy: Mapping[str, object],
|
||||||
|
*,
|
||||||
|
schema: Sequence[DatasourceField] = (),
|
||||||
|
) -> Mapping[str, object]:
|
||||||
|
normalized = _normalize_policy(policy)
|
||||||
|
if schema:
|
||||||
|
known = {field.name for field in schema}
|
||||||
|
configured = set(cast(Mapping[str, object], normalized.get("fields", {})))
|
||||||
|
configured.update(
|
||||||
|
str(item["field"])
|
||||||
|
for item in cast(
|
||||||
|
Sequence[Mapping[str, object]], normalized.get("row_filters", ())
|
||||||
|
)
|
||||||
|
)
|
||||||
|
unknown = sorted(configured - known)
|
||||||
|
if unknown:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Visibility policy references unknown fields: {', '.join(unknown)}."
|
||||||
|
)
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
|
def visible_schema(
|
||||||
|
schema: Sequence[DatasourceField],
|
||||||
|
plan: VisibilityPlan,
|
||||||
|
) -> tuple[DatasourceField, ...]:
|
||||||
|
classifications: dict[str, str] = {}
|
||||||
|
omitted: set[str] = set()
|
||||||
|
for rule in plan.field_rules:
|
||||||
|
classifications[rule.name] = rule.classification
|
||||||
|
if not rule.allowed and rule.action == "omit":
|
||||||
|
omitted.add(rule.name)
|
||||||
|
return tuple(
|
||||||
|
DatasourceField(
|
||||||
|
name=field.name,
|
||||||
|
data_type=field.data_type,
|
||||||
|
nullable=field.nullable,
|
||||||
|
classification=classifications.get(field.name, field.classification),
|
||||||
|
)
|
||||||
|
for field in schema
|
||||||
|
if field.name not in omitted
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def apply_visibility(
|
||||||
|
rows: Sequence[Mapping[str, object]],
|
||||||
|
*,
|
||||||
|
principal: PrincipalRef,
|
||||||
|
plan: VisibilityPlan,
|
||||||
|
columns: Sequence[str] = (),
|
||||||
|
) -> tuple[tuple[Mapping[str, object], ...], int]:
|
||||||
|
omitted = {
|
||||||
|
rule.name
|
||||||
|
for rule in plan.field_rules
|
||||||
|
if not rule.allowed and rule.action == "omit"
|
||||||
|
}
|
||||||
|
redacted = {
|
||||||
|
rule.name
|
||||||
|
for rule in plan.field_rules
|
||||||
|
if not rule.allowed and rule.action == "redact"
|
||||||
|
}
|
||||||
|
requested = tuple(dict.fromkeys(columns))
|
||||||
|
result: list[Mapping[str, object]] = []
|
||||||
|
filtered = 0
|
||||||
|
for source_row in rows:
|
||||||
|
if not all(
|
||||||
|
_row_filter_allows(source_row, item, principal) for item in plan.row_filters
|
||||||
|
):
|
||||||
|
filtered += 1
|
||||||
|
continue
|
||||||
|
names = requested or tuple(source_row)
|
||||||
|
result.append(
|
||||||
|
{
|
||||||
|
name: None if name in redacted else source_row.get(name)
|
||||||
|
for name in names
|
||||||
|
if name not in omitted
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return tuple(result), filtered
|
||||||
|
|
||||||
|
|
||||||
|
def visibility_diagnostics(
|
||||||
|
plan: VisibilityPlan,
|
||||||
|
*,
|
||||||
|
scanned_rows: int,
|
||||||
|
permitted_rows: int,
|
||||||
|
filtered_rows: int,
|
||||||
|
scan_limited: bool,
|
||||||
|
) -> tuple[TabularPreviewDiagnostic, ...]:
|
||||||
|
if not plan.applied:
|
||||||
|
return ()
|
||||||
|
diagnostics = [
|
||||||
|
TabularPreviewDiagnostic(
|
||||||
|
severity="info",
|
||||||
|
code="datasource.visibility_applied",
|
||||||
|
message="Datasource visibility policy was applied before rows left the provider.",
|
||||||
|
details={
|
||||||
|
"policy_count": len(plan.policy_hashes),
|
||||||
|
"field_rule_count": len(plan.field_rules),
|
||||||
|
"row_filter_count": len(plan.row_filters),
|
||||||
|
"scanned_rows": scanned_rows,
|
||||||
|
"permitted_rows": permitted_rows,
|
||||||
|
"filtered_rows": filtered_rows,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
]
|
||||||
|
if scan_limited:
|
||||||
|
diagnostics.append(
|
||||||
|
TabularPreviewDiagnostic(
|
||||||
|
severity="warning",
|
||||||
|
code="datasource.visibility_scan_limited",
|
||||||
|
message="The bounded visibility scan ended before the origin was exhausted.",
|
||||||
|
details={"scanned_rows": scanned_rows},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return tuple(diagnostics)
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_policy(policy: Mapping[str, object]) -> dict[str, object]:
|
||||||
|
unknown = set(policy) - _POLICY_KEYS
|
||||||
|
if unknown:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Unsupported visibility policy keys: {', '.join(sorted(unknown))}."
|
||||||
|
)
|
||||||
|
result: dict[str, object] = {}
|
||||||
|
for name in ("source_acl", "materialization_acl"):
|
||||||
|
if name in policy:
|
||||||
|
result[name] = _normalize_acl(policy[name], path=name)
|
||||||
|
|
||||||
|
raw_fields = policy.get("fields", {})
|
||||||
|
if not isinstance(raw_fields, Mapping) or len(raw_fields) > 500:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"Visibility policy fields must be a mapping of at most 500 fields."
|
||||||
|
)
|
||||||
|
fields: dict[str, object] = {}
|
||||||
|
for raw_name, raw_rule in sorted(raw_fields.items(), key=lambda item: str(item[0])):
|
||||||
|
name = str(raw_name).strip()
|
||||||
|
if not name or not isinstance(raw_rule, Mapping):
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"Every visibility field rule needs a field name and mapping."
|
||||||
|
)
|
||||||
|
unknown_rule = set(raw_rule) - {"classification", "action", "allow"}
|
||||||
|
if unknown_rule:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Unsupported visibility field keys for {name!r}: {', '.join(sorted(unknown_rule))}."
|
||||||
|
)
|
||||||
|
classification = str(raw_rule.get("classification") or "restricted").strip()
|
||||||
|
if not classification or len(classification) > 80:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"Field classifications must contain 1 to 80 characters."
|
||||||
|
)
|
||||||
|
action = str(raw_rule.get("action") or "redact").strip()
|
||||||
|
if action not in {"redact", "omit"}:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"Field visibility action must be redact or omit."
|
||||||
|
)
|
||||||
|
if "allow" not in raw_rule:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Visibility field rule {name!r} needs an allow ACL."
|
||||||
|
)
|
||||||
|
fields[name] = {
|
||||||
|
"classification": classification,
|
||||||
|
"action": action,
|
||||||
|
"allow": _normalize_acl(raw_rule["allow"], path=f"fields.{name}.allow"),
|
||||||
|
}
|
||||||
|
if fields:
|
||||||
|
result["fields"] = fields
|
||||||
|
|
||||||
|
raw_filters = policy.get("row_filters", ())
|
||||||
|
if not isinstance(raw_filters, Sequence) or isinstance(raw_filters, (str, bytes)):
|
||||||
|
raise DatasourceValidationError("Visibility row_filters must be a list.")
|
||||||
|
if len(raw_filters) > 50:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"Visibility policy supports at most 50 row filters."
|
||||||
|
)
|
||||||
|
filters: list[dict[str, object]] = []
|
||||||
|
for raw_filter in raw_filters:
|
||||||
|
if not isinstance(raw_filter, Mapping):
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"Every visibility row filter must be a mapping."
|
||||||
|
)
|
||||||
|
unknown_filter = set(raw_filter) - {"field", "claim", "operator", "allow_null"}
|
||||||
|
if unknown_filter:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Unsupported row-filter keys: {', '.join(sorted(unknown_filter))}."
|
||||||
|
)
|
||||||
|
field = str(raw_filter.get("field") or "").strip()
|
||||||
|
claim = str(raw_filter.get("claim") or "").strip()
|
||||||
|
operator = str(raw_filter.get("operator") or "equals").strip()
|
||||||
|
if not field:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
"Every visibility row filter needs a field."
|
||||||
|
)
|
||||||
|
if claim not in _CLAIM_KEYS:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Unsupported visibility row-filter claim {claim!r}."
|
||||||
|
)
|
||||||
|
if operator not in {"equals", "in"}:
|
||||||
|
raise DatasourceValidationError("Row-filter operator must be equals or in.")
|
||||||
|
filters.append(
|
||||||
|
{
|
||||||
|
"field": field,
|
||||||
|
"claim": claim,
|
||||||
|
"operator": operator,
|
||||||
|
"allow_null": bool(raw_filter.get("allow_null", False)),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
if filters:
|
||||||
|
result["row_filters"] = filters
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_acl(value: object, *, path: str) -> dict[str, list[str]]:
|
||||||
|
if not isinstance(value, Mapping):
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Visibility policy {path} must be an ACL mapping."
|
||||||
|
)
|
||||||
|
unknown = set(value) - _ACL_KEYS
|
||||||
|
if unknown:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Unsupported ACL selectors in {path}: {', '.join(sorted(unknown))}."
|
||||||
|
)
|
||||||
|
result: dict[str, list[str]] = {}
|
||||||
|
for key in sorted(value):
|
||||||
|
raw_values = value[key]
|
||||||
|
if not isinstance(raw_values, Sequence) or isinstance(raw_values, (str, bytes)):
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Visibility ACL selector {path}.{key} must be a list."
|
||||||
|
)
|
||||||
|
if len(raw_values) > 250:
|
||||||
|
raise DatasourceValidationError(
|
||||||
|
f"Visibility ACL selector {path}.{key} is limited to 250 entries."
|
||||||
|
)
|
||||||
|
normalized = sorted(
|
||||||
|
{str(item).strip() for item in raw_values if str(item).strip()}
|
||||||
|
)
|
||||||
|
result[key] = normalized
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _acl_allows(acl: Mapping[str, object], principal: PrincipalRef) -> bool:
|
||||||
|
facts = _principal_policy_facts(principal)
|
||||||
|
return any(set(values) & set(facts.get(key, ())) for key, values in acl.items())
|
||||||
|
|
||||||
|
|
||||||
|
def _principal_policy_facts(principal: PrincipalRef) -> dict[str, tuple[str, ...]]:
|
||||||
|
return {
|
||||||
|
"account_ids": _optional_tuple(principal.account_id),
|
||||||
|
"membership_ids": _optional_tuple(principal.membership_id),
|
||||||
|
"identity_ids": _optional_tuple(principal.identity_id),
|
||||||
|
"group_ids": tuple(sorted(principal.group_ids)),
|
||||||
|
"role_ids": tuple(sorted(principal.role_ids)),
|
||||||
|
"service_account_ids": _optional_tuple(principal.service_account_id),
|
||||||
|
"auth_methods": (principal.auth_method,),
|
||||||
|
"function_assignment_ids": tuple(sorted(principal.function_assignment_ids)),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _claim_values(principal: PrincipalRef, claim: str) -> tuple[str, ...]:
|
||||||
|
facts = _principal_policy_facts(principal)
|
||||||
|
aliases = {
|
||||||
|
"account_id": "account_ids",
|
||||||
|
"membership_id": "membership_ids",
|
||||||
|
"identity_id": "identity_ids",
|
||||||
|
"service_account_id": "service_account_ids",
|
||||||
|
}
|
||||||
|
return facts.get(aliases.get(claim, claim), ())
|
||||||
|
|
||||||
|
|
||||||
|
def _relevant_policy_fact_keys(
|
||||||
|
policies: Sequence[Mapping[str, object]],
|
||||||
|
) -> set[str]:
|
||||||
|
keys: set[str] = set()
|
||||||
|
aliases = {
|
||||||
|
"account_id": "account_ids",
|
||||||
|
"membership_id": "membership_ids",
|
||||||
|
"identity_id": "identity_ids",
|
||||||
|
"service_account_id": "service_account_ids",
|
||||||
|
}
|
||||||
|
for policy in policies:
|
||||||
|
for acl_name in ("source_acl", "materialization_acl"):
|
||||||
|
acl = policy.get(acl_name)
|
||||||
|
if isinstance(acl, Mapping):
|
||||||
|
keys.update(str(key) for key in acl)
|
||||||
|
fields = policy.get("fields")
|
||||||
|
if isinstance(fields, Mapping):
|
||||||
|
for rule in fields.values():
|
||||||
|
if isinstance(rule, Mapping) and isinstance(rule.get("allow"), Mapping):
|
||||||
|
keys.update(str(key) for key in rule["allow"])
|
||||||
|
for row_filter in policy.get("row_filters", ()):
|
||||||
|
if isinstance(row_filter, Mapping):
|
||||||
|
claim = str(row_filter.get("claim") or "")
|
||||||
|
keys.add(aliases.get(claim, claim))
|
||||||
|
return keys
|
||||||
|
|
||||||
|
|
||||||
|
def _row_filter_allows(
|
||||||
|
row: Mapping[str, object],
|
||||||
|
item: RowFilter,
|
||||||
|
principal: PrincipalRef,
|
||||||
|
) -> bool:
|
||||||
|
value = row.get(item.field)
|
||||||
|
if value is None:
|
||||||
|
return item.allow_null
|
||||||
|
claims = set(_claim_values(principal, item.claim))
|
||||||
|
if not claims:
|
||||||
|
return False
|
||||||
|
if isinstance(value, Sequence) and not isinstance(value, (str, bytes)):
|
||||||
|
values = {str(candidate) for candidate in value}
|
||||||
|
return bool(values & claims)
|
||||||
|
return str(value) in claims
|
||||||
|
|
||||||
|
|
||||||
|
def _optional_tuple(value: object | None) -> tuple[str, ...]:
|
||||||
|
return (str(value),) if value is not None and str(value) else ()
|
||||||
|
|
||||||
|
|
||||||
|
def _digest(value: object) -> str:
|
||||||
|
encoded = json.dumps(value, sort_keys=True, separators=(",", ":"), default=str)
|
||||||
|
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"VisibilityPlan",
|
||||||
|
"apply_visibility",
|
||||||
|
"normalize_visibility_policy",
|
||||||
|
"visibility_diagnostics",
|
||||||
|
"visibility_plan",
|
||||||
|
"visible_schema",
|
||||||
|
]
|
||||||
@@ -20,6 +20,7 @@ from govoplan_core.privacy.dsar_workflow import (
|
|||||||
)
|
)
|
||||||
from govoplan_datasources.backend.db.models import (
|
from govoplan_datasources.backend.db.models import (
|
||||||
DatasourceGovernanceReferenceRecord,
|
DatasourceGovernanceReferenceRecord,
|
||||||
|
DatasourceLifecycleEvidenceRecord,
|
||||||
DatasourceMaterializationRecord,
|
DatasourceMaterializationRecord,
|
||||||
DatasourcePayloadRecord,
|
DatasourcePayloadRecord,
|
||||||
DatasourcePayloadRowRecord,
|
DatasourcePayloadRowRecord,
|
||||||
@@ -382,6 +383,54 @@ class DatasourcesDsarProviderTests(unittest.TestCase):
|
|||||||
self.assertEqual((), wrong_tenant)
|
self.assertEqual((), wrong_tenant)
|
||||||
self.assertEqual((), conflict)
|
self.assertEqual((), conflict)
|
||||||
|
|
||||||
|
def test_lifecycle_evidence_is_minimized_and_immutable(self) -> None:
|
||||||
|
evidence = DatasourceLifecycleEvidenceRecord(
|
||||||
|
id="evidence-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
subject_ref="datasource:datasource-1",
|
||||||
|
event_type="retention.applied",
|
||||||
|
occurred_at=NOW,
|
||||||
|
actor_ref="account-1",
|
||||||
|
policy_version="retention-v1",
|
||||||
|
policy_hash="1" * 64,
|
||||||
|
subject_digest="2" * 64,
|
||||||
|
event_hash="3" * 64,
|
||||||
|
details_={"secret": SECRET},
|
||||||
|
)
|
||||||
|
self.session.add(evidence)
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
exact_subject = DsarSubjectRef(
|
||||||
|
external_references={
|
||||||
|
"datasources.lifecycle_evidence": "evidence-1",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
exact = self.provider.search_subject(
|
||||||
|
self.session,
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
subject=exact_subject,
|
||||||
|
)
|
||||||
|
self.assertEqual(1, len(exact))
|
||||||
|
self.assertEqual("datasource_operator_attribution", exact[0].category)
|
||||||
|
self.assertNotIn(SECRET, json.dumps(exact[0].to_dict()))
|
||||||
|
actions = self.provider.plan_erasure(
|
||||||
|
self.session,
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
subject=exact_subject,
|
||||||
|
records=exact,
|
||||||
|
)
|
||||||
|
self.assertEqual({"retain"}, {action.kind for action in actions})
|
||||||
|
|
||||||
|
canonical = self.provider.search_subject(
|
||||||
|
self.session,
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
subject=DsarSubjectRef(account_id="account-1"),
|
||||||
|
)
|
||||||
|
self.assertIn(
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
|
{record.resource_type for record in canonical},
|
||||||
|
)
|
||||||
|
|
||||||
def test_transient_deletion_is_safe_and_idempotent(self) -> None:
|
def test_transient_deletion_is_safe_and_idempotent(self) -> None:
|
||||||
subject = DsarSubjectRef(
|
subject = DsarSubjectRef(
|
||||||
external_references={
|
external_references={
|
||||||
|
|||||||
@@ -6,6 +6,14 @@ from govoplan_datasources.backend.manifest import manifest
|
|||||||
|
|
||||||
|
|
||||||
class DatasourcesInterfaceDocumentationContractTests(unittest.TestCase):
|
class DatasourcesInterfaceDocumentationContractTests(unittest.TestCase):
|
||||||
|
def test_all_static_topics_have_complete_german_content(self) -> None:
|
||||||
|
for topic in manifest.documentation:
|
||||||
|
german = (topic.translations or {}).get("de", {})
|
||||||
|
self.assertEqual({"title", "summary", "body"}, set(german), topic.id)
|
||||||
|
self.assertTrue(
|
||||||
|
all(str(value).strip() for value in german.values()), topic.id
|
||||||
|
)
|
||||||
|
|
||||||
def test_route_and_surfaces_remain_declared(self) -> None:
|
def test_route_and_surfaces_remain_declared(self) -> None:
|
||||||
frontend = manifest.frontend
|
frontend = manifest.frontend
|
||||||
self.assertIsNotNone(frontend)
|
self.assertIsNotNone(frontend)
|
||||||
@@ -18,6 +26,8 @@ class DatasourcesInterfaceDocumentationContractTests(unittest.TestCase):
|
|||||||
"datasources.origins",
|
"datasources.origins",
|
||||||
"datasources.governance",
|
"datasources.governance",
|
||||||
"datasources.preview",
|
"datasources.preview",
|
||||||
|
"datasources.lifecycle-evidence",
|
||||||
|
"datasources.retention",
|
||||||
},
|
},
|
||||||
{item.id for item in frontend.view_surfaces}, # type: ignore[union-attr]
|
{item.id for item in frontend.view_surfaces}, # type: ignore[union-attr]
|
||||||
)
|
)
|
||||||
@@ -27,16 +37,33 @@ class DatasourcesInterfaceDocumentationContractTests(unittest.TestCase):
|
|||||||
lifecycle = topics["datasources.lifecycle"]
|
lifecycle = topics["datasources.lifecycle"]
|
||||||
governance = topics["datasources.governance"]
|
governance = topics["datasources.governance"]
|
||||||
quality = topics["datasources.quality-gates"]
|
quality = topics["datasources.quality-gates"]
|
||||||
|
visibility = topics["datasources.visibility"]
|
||||||
reference = topics["datasources.reference.fields-and-consequences"]
|
reference = topics["datasources.reference.fields-and-consequences"]
|
||||||
|
lifecycle_controls = topics["datasources.approval-and-retention"]
|
||||||
|
|
||||||
self.assertIn("datasources.staging", lifecycle.metadata["help_contexts"])
|
self.assertIn("datasources.staging", lifecycle.metadata["help_contexts"])
|
||||||
self.assertIn("datasources.field.authority-mode", governance.metadata["help_contexts"])
|
self.assertIn(
|
||||||
self.assertIn("datasources.staging.validation", quality.metadata["help_contexts"])
|
"datasources.field.authority-mode", governance.metadata["help_contexts"]
|
||||||
|
)
|
||||||
|
self.assertIn(
|
||||||
|
"datasources.staging.validation", quality.metadata["help_contexts"]
|
||||||
|
)
|
||||||
|
self.assertIn(
|
||||||
|
"datasources.field.visibility-policy",
|
||||||
|
visibility.metadata["help_contexts"],
|
||||||
|
)
|
||||||
|
self.assertIn("fails closed", visibility.body)
|
||||||
self.assertIn("policy version and hash", quality.body)
|
self.assertIn("policy version and hash", quality.body)
|
||||||
self.assertTrue(quality.metadata["limitations"])
|
self.assertTrue(quality.metadata["limitations"])
|
||||||
self.assertIn("datasources.action.promote", reference.metadata["help_contexts"])
|
self.assertIn("datasources.action.promote", reference.metadata["help_contexts"])
|
||||||
self.assertIn("freeze", reference.metadata["consequence_classes"])
|
self.assertIn("freeze", reference.metadata["consequence_classes"])
|
||||||
self.assertIn("retire", reference.metadata["consequence_classes"])
|
self.assertIn("retire", reference.metadata["consequence_classes"])
|
||||||
|
self.assertEqual("reference", reference.metadata["kind"])
|
||||||
|
self.assertIn(
|
||||||
|
"datasources.action.approve", lifecycle_controls.metadata["help_contexts"]
|
||||||
|
)
|
||||||
|
self.assertIn("hash-chained lifecycle evidence", lifecycle_controls.body)
|
||||||
|
self.assertTrue(lifecycle_controls.translations["de"]["body"])
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -1,6 +1,10 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import unittest
|
import unittest
|
||||||
|
from dataclasses import replace
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from fastapi import HTTPException
|
||||||
|
|
||||||
from sqlalchemy import create_engine
|
from sqlalchemy import create_engine
|
||||||
from sqlalchemy.orm import sessionmaker
|
from sqlalchemy.orm import sessionmaker
|
||||||
@@ -24,6 +28,9 @@ from govoplan_core.core.datasources import (
|
|||||||
DatasourceValidationError,
|
DatasourceValidationError,
|
||||||
)
|
)
|
||||||
from govoplan_core.core.tabular_sources import (
|
from govoplan_core.core.tabular_sources import (
|
||||||
|
TabularCsvSource,
|
||||||
|
parse_tabular_csv,
|
||||||
|
csv_source_payload,
|
||||||
TabularPreviewDiagnostic,
|
TabularPreviewDiagnostic,
|
||||||
TabularPushdown,
|
TabularPushdown,
|
||||||
TabularSourceHealth,
|
TabularSourceHealth,
|
||||||
@@ -31,6 +38,7 @@ from govoplan_core.core.tabular_sources import (
|
|||||||
from govoplan_core.db.base import Base, utcnow
|
from govoplan_core.db.base import Base, utcnow
|
||||||
from govoplan_datasources.backend.db.models import (
|
from govoplan_datasources.backend.db.models import (
|
||||||
DatasourceGovernanceReferenceRecord,
|
DatasourceGovernanceReferenceRecord,
|
||||||
|
DatasourceLifecycleEvidenceRecord,
|
||||||
DatasourceMaterializationRecord,
|
DatasourceMaterializationRecord,
|
||||||
DatasourcePayloadRecord,
|
DatasourcePayloadRecord,
|
||||||
DatasourcePayloadRowRecord,
|
DatasourcePayloadRowRecord,
|
||||||
@@ -39,6 +47,7 @@ from govoplan_datasources.backend.db.models import (
|
|||||||
DatasourceStageRecord,
|
DatasourceStageRecord,
|
||||||
)
|
)
|
||||||
from govoplan_datasources.backend.service import (
|
from govoplan_datasources.backend.service import (
|
||||||
|
ADMIN_SCOPE,
|
||||||
CATALOGUE_READ_SCOPE,
|
CATALOGUE_READ_SCOPE,
|
||||||
SOURCE_WRITE_SCOPE,
|
SOURCE_WRITE_SCOPE,
|
||||||
STAGE_WRITE_SCOPE,
|
STAGE_WRITE_SCOPE,
|
||||||
@@ -226,6 +235,7 @@ class DatasourceLifecycleTests(unittest.TestCase):
|
|||||||
tables=[
|
tables=[
|
||||||
DatasourceRecord.__table__,
|
DatasourceRecord.__table__,
|
||||||
DatasourceGovernanceReferenceRecord.__table__,
|
DatasourceGovernanceReferenceRecord.__table__,
|
||||||
|
DatasourceLifecycleEvidenceRecord.__table__,
|
||||||
DatasourcePayloadRecord.__table__,
|
DatasourcePayloadRecord.__table__,
|
||||||
DatasourcePayloadRowRecord.__table__,
|
DatasourcePayloadRowRecord.__table__,
|
||||||
DatasourceMaterializationRecord.__table__,
|
DatasourceMaterializationRecord.__table__,
|
||||||
@@ -248,6 +258,7 @@ class DatasourceLifecycleTests(unittest.TestCase):
|
|||||||
Base.metadata.drop_all(
|
Base.metadata.drop_all(
|
||||||
self.engine,
|
self.engine,
|
||||||
tables=[
|
tables=[
|
||||||
|
DatasourceLifecycleEvidenceRecord.__table__,
|
||||||
DatasourceStageRecord.__table__,
|
DatasourceStageRecord.__table__,
|
||||||
DatasourcePublicationRecord.__table__,
|
DatasourcePublicationRecord.__table__,
|
||||||
DatasourceMaterializationRecord.__table__,
|
DatasourceMaterializationRecord.__table__,
|
||||||
@@ -260,6 +271,85 @@ class DatasourceLifecycleTests(unittest.TestCase):
|
|||||||
)
|
)
|
||||||
self.engine.dispose()
|
self.engine.dispose()
|
||||||
|
|
||||||
|
def test_original_csv_preserves_source_but_respects_current_and_historical_visibility(self) -> None:
|
||||||
|
text = 'value\r\n" keep me "\r\n0.123456789012345678901234567890\r\n" "\r\n'
|
||||||
|
source = TabularCsvSource(text=text, value_mode="text")
|
||||||
|
request = DatasourceStageInput(name="Text", source_name="csv_text", kind="upload", mode="static", shape="tabular", rows=parse_tabular_csv(text, value_mode="text"), csv_source=source)
|
||||||
|
with self.assertRaises(DatasourceValidationError):
|
||||||
|
self.provider.create_stage(self.session, principal(), stage=replace(request, rows=({"value": "changed"},)))
|
||||||
|
stage = self.provider.create_stage(self.session, principal(), stage=request)
|
||||||
|
descriptor, materialization = self.provider.promote_stage(self.session, principal(), stage_ref=stage.ref)
|
||||||
|
self.session.commit()
|
||||||
|
self.session.expunge_all()
|
||||||
|
admin = principal(scopes=(ADMIN_SCOPE,))
|
||||||
|
self.assertEqual(text, self.provider.original_csv(self.session, admin, datasource_ref=descriptor.ref, materialization_ref=materialization.ref))
|
||||||
|
from govoplan_datasources.backend.router import api_original_csv
|
||||||
|
with patch("govoplan_datasources.backend.router._provider", return_value=self.provider), patch("govoplan_datasources.backend.router._audit") as audit:
|
||||||
|
response = api_original_csv(descriptor.ref.split(":", 1)[1], materialization.ref.split(":", 1)[1], session=self.session, principal=admin)
|
||||||
|
self.assertEqual(text.encode("utf-8"), response.body)
|
||||||
|
self.assertEqual("no-store", response.headers["cache-control"])
|
||||||
|
self.assertEqual("datasources.original_csv.exported", audit.call_args.kwargs["action"])
|
||||||
|
self.assertNotIn(text, repr(audit.call_args.kwargs["details"]))
|
||||||
|
with patch("govoplan_datasources.backend.router._provider") as read, self.assertRaises(HTTPException) as denied:
|
||||||
|
api_original_csv("anything", "anything", session=self.session, principal=principal())
|
||||||
|
self.assertEqual(403, denied.exception.status_code)
|
||||||
|
read.assert_not_called()
|
||||||
|
frozen = self.provider.freeze_datasource(self.session, admin, datasource_ref=descriptor.ref)
|
||||||
|
self.assertEqual(text, self.provider.original_csv(self.session, admin, datasource_ref=descriptor.ref, materialization_ref=frozen.ref))
|
||||||
|
self.assertNotIn("text", descriptor.metadata["csv_source"])
|
||||||
|
self.assertNotIn("text", materialization.metadata["csv_source"])
|
||||||
|
with self.assertRaises(DatasourceAccessError):
|
||||||
|
self.provider.original_csv(self.session, principal(), datasource_ref=descriptor.ref, materialization_ref=materialization.ref)
|
||||||
|
from govoplan_core.core.datasources import DatasourceNotFoundError
|
||||||
|
with self.assertRaises(DatasourceNotFoundError):
|
||||||
|
self.provider.original_csv(self.session, principal("other-tenant", scopes=(ADMIN_SCOPE,)), datasource_ref=descriptor.ref, materialization_ref=materialization.ref)
|
||||||
|
record = self.session.get(DatasourceRecord, descriptor.ref.split(":", 1)[1])
|
||||||
|
record.visibility_policy = {"fields": {"value": {"classification": "restricted", "action": "omit", "allow": {"account_ids": ["other"]}}}}
|
||||||
|
self.session.flush()
|
||||||
|
with self.assertRaises(DatasourceAccessError):
|
||||||
|
self.provider.original_csv(self.session, admin, datasource_ref=descriptor.ref, materialization_ref=materialization.ref)
|
||||||
|
record.visibility_policy = {}
|
||||||
|
stored = self.session.get(DatasourceMaterializationRecord, materialization.ref.split(":", 1)[1])
|
||||||
|
stored.governance_snapshot_ = {**stored.governance_snapshot_, "visibility_policy": {"row_filters": [{"field": "value", "claim": "account_id"}]}}
|
||||||
|
self.session.flush()
|
||||||
|
with self.assertRaises(DatasourceAccessError):
|
||||||
|
self.provider.original_csv(self.session, admin, datasource_ref=descriptor.ref, materialization_ref=materialization.ref)
|
||||||
|
stored.governance_snapshot_ = {**stored.governance_snapshot_, "visibility_policy": {}}
|
||||||
|
stored.csv_source_ = {**stored.csv_source_, "text": "changed"}
|
||||||
|
self.session.flush()
|
||||||
|
with self.assertRaises(DatasourceUnavailableError):
|
||||||
|
self.provider.original_csv(self.session, admin, datasource_ref=descriptor.ref, materialization_ref=materialization.ref)
|
||||||
|
|
||||||
|
def test_csv_stage_projection_binding_rejects_equal_but_different_types(self) -> None:
|
||||||
|
for text, value in (("value\ntrue\n", 1), ("value\n1\n", True), ("value\n1\n", 1.0)):
|
||||||
|
with self.subTest(text=text, value=value), self.assertRaises(DatasourceValidationError):
|
||||||
|
self.provider.create_stage(self.session, principal(), stage=DatasourceStageInput(name="Invalid", source_name="invalid", kind="upload", mode="static", shape="tabular", rows=({"value": value},), csv_source=TabularCsvSource(text=text)))
|
||||||
|
|
||||||
|
def test_csv_invalid_unicode_is_validation_in_both_modes(self) -> None:
|
||||||
|
from govoplan_datasources.backend.tabular import parse_csv_rows
|
||||||
|
for mode in ("text", "legacy_typed"):
|
||||||
|
with self.subTest(mode=mode), self.assertRaisesRegex(DatasourceValidationError, "valid Unicode"):
|
||||||
|
parse_csv_rows("value\nprivate-\ud800\n", value_mode=mode)
|
||||||
|
|
||||||
|
def test_freezing_csv_preserves_historical_restrictions_and_rejects_resealed_drift(self) -> None:
|
||||||
|
source = TabularCsvSource(text='value\nprivate\n', value_mode="text")
|
||||||
|
governance = DatasourceGovernance(visibility_policy={"fields": {"value": {"classification": "restricted", "action": "omit", "allow": {"account_ids": ["other"]}}}})
|
||||||
|
stage = self.provider.create_stage(self.session, principal(), stage=DatasourceStageInput(name="Private CSV", source_name="private_csv", kind="upload", mode="static", shape="tabular", rows=parse_tabular_csv(source.text, value_mode="text"), csv_source=source, governance=governance))
|
||||||
|
descriptor, original = self.provider.promote_stage(self.session, principal(), stage_ref=stage.ref)
|
||||||
|
record = self.session.get(DatasourceRecord, descriptor.ref.split(":", 1)[1])
|
||||||
|
record.visibility_policy = {}
|
||||||
|
self.session.flush()
|
||||||
|
admin = principal(scopes=(ADMIN_SCOPE,))
|
||||||
|
frozen = self.provider.freeze_datasource(self.session, admin, datasource_ref=descriptor.ref)
|
||||||
|
for ref in (original.ref, frozen.ref):
|
||||||
|
with self.subTest(ref=ref), self.assertRaises(DatasourceAccessError):
|
||||||
|
self.provider.original_csv(self.session, admin, datasource_ref=descriptor.ref, materialization_ref=ref)
|
||||||
|
stored = self.session.get(DatasourceMaterializationRecord, original.ref.split(":", 1)[1])
|
||||||
|
stored.csv_source_ = csv_source_payload(TabularCsvSource(text="value\nchanged\n", value_mode="text"))
|
||||||
|
self.session.flush()
|
||||||
|
with self.assertRaises(DatasourceUnavailableError):
|
||||||
|
self.provider.freeze_datasource(self.session, admin, datasource_ref=descriptor.ref)
|
||||||
|
|
||||||
def test_static_stage_promote_update_and_frozen_read(self) -> None:
|
def test_static_stage_promote_update_and_frozen_read(self) -> None:
|
||||||
first_stage = self.provider.create_stage(
|
first_stage = self.provider.create_stage(
|
||||||
self.session,
|
self.session,
|
||||||
|
|||||||
@@ -0,0 +1,503 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from govoplan_core.core.tabular_sources import TabularCsvSource
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
from datetime import timedelta
|
||||||
|
|
||||||
|
from sqlalchemy import create_engine, select
|
||||||
|
from sqlalchemy.orm import sessionmaker
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal
|
||||||
|
from govoplan_core.core.access import PrincipalRef
|
||||||
|
from govoplan_core.core.datasources import (
|
||||||
|
CAPABILITY_DATASOURCE_ORIGINS,
|
||||||
|
DatasourceField,
|
||||||
|
DatasourceGovernance,
|
||||||
|
DatasourceOrigin,
|
||||||
|
DatasourceOriginReadRequest,
|
||||||
|
DatasourceOriginReadResult,
|
||||||
|
DatasourceReadRequest,
|
||||||
|
DatasourceStageInput,
|
||||||
|
DatasourceUnavailableError,
|
||||||
|
DatasourceValidationError,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.tabular_sources import TabularPushdown, TabularSourceHealth
|
||||||
|
from govoplan_core.db.base import Base
|
||||||
|
from govoplan_datasources.backend.db.models import (
|
||||||
|
DatasourceGovernanceReferenceRecord,
|
||||||
|
DatasourceLifecycleEvidenceRecord,
|
||||||
|
DatasourceMaterializationRecord,
|
||||||
|
DatasourcePayloadRecord,
|
||||||
|
DatasourcePayloadRowRecord,
|
||||||
|
DatasourcePublicationRecord,
|
||||||
|
DatasourceRecord,
|
||||||
|
DatasourceStageRecord,
|
||||||
|
)
|
||||||
|
from govoplan_datasources.backend.service import (
|
||||||
|
CATALOGUE_READ_SCOPE,
|
||||||
|
SOURCE_WRITE_SCOPE,
|
||||||
|
STAGE_APPROVE_SCOPE,
|
||||||
|
STAGE_WRITE_SCOPE,
|
||||||
|
SqlDatasourceProvider,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def principal(
|
||||||
|
account_id: str,
|
||||||
|
*,
|
||||||
|
scopes: tuple[str, ...],
|
||||||
|
) -> ApiPrincipal:
|
||||||
|
return ApiPrincipal(
|
||||||
|
principal=PrincipalRef(
|
||||||
|
account_id=account_id,
|
||||||
|
membership_id=f"membership-{account_id}",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
scopes=frozenset(scopes),
|
||||||
|
),
|
||||||
|
account=object(),
|
||||||
|
user=object(),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class _OriginProvider:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.rows = [{"id": 1, "status": "initial"}]
|
||||||
|
|
||||||
|
def origin(self) -> DatasourceOrigin:
|
||||||
|
return DatasourceOrigin(
|
||||||
|
ref="origin:cases",
|
||||||
|
source_name="cached_cases",
|
||||||
|
name="Cached cases",
|
||||||
|
kind="database",
|
||||||
|
shape="tabular",
|
||||||
|
supported_modes=("cached",),
|
||||||
|
provider="connectors.test",
|
||||||
|
schema=(
|
||||||
|
DatasourceField("id", "integer", False),
|
||||||
|
DatasourceField("status", "string", False),
|
||||||
|
),
|
||||||
|
schema_version="1",
|
||||||
|
fingerprint=f"cases-{self.rows[-1]['status']}",
|
||||||
|
row_count=len(self.rows),
|
||||||
|
source_mode="cached",
|
||||||
|
pushdown=TabularPushdown(pagination=True),
|
||||||
|
health=TabularSourceHealth(
|
||||||
|
status="healthy",
|
||||||
|
code="origin.ready",
|
||||||
|
summary="Origin is ready.",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def list_origins(self, _session, _principal, *, query="", limit=100):
|
||||||
|
del limit
|
||||||
|
origin = self.origin()
|
||||||
|
return (origin,) if query.casefold() in origin.name.casefold() else ()
|
||||||
|
|
||||||
|
def get_origin(self, _session, _principal, *, origin_ref):
|
||||||
|
return self.origin() if origin_ref == "origin:cases" else None
|
||||||
|
|
||||||
|
def read_origin(
|
||||||
|
self,
|
||||||
|
_session,
|
||||||
|
_principal,
|
||||||
|
*,
|
||||||
|
request: DatasourceOriginReadRequest,
|
||||||
|
) -> DatasourceOriginReadResult:
|
||||||
|
rows = self.rows[request.offset : request.offset + request.limit]
|
||||||
|
return DatasourceOriginReadResult(
|
||||||
|
origin=self.origin(),
|
||||||
|
rows=tuple(dict(row) for row in rows),
|
||||||
|
total_rows=len(self.rows),
|
||||||
|
truncated=request.offset + len(rows) < len(self.rows),
|
||||||
|
returned_bytes=64,
|
||||||
|
elapsed_ms=1,
|
||||||
|
effective_row_limit=request.limit,
|
||||||
|
effective_byte_limit=request.max_bytes,
|
||||||
|
effective_timeout_ms=request.timeout_ms,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class _Registry:
|
||||||
|
def __init__(self, origins: _OriginProvider) -> None:
|
||||||
|
self.origins = origins
|
||||||
|
|
||||||
|
def has_capability(self, name: str) -> bool:
|
||||||
|
return name == CAPABILITY_DATASOURCE_ORIGINS
|
||||||
|
|
||||||
|
def capability(self, name: str):
|
||||||
|
if not self.has_capability(name):
|
||||||
|
raise KeyError(name)
|
||||||
|
return self.origins
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceLifecycleGovernanceTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.engine = create_engine("sqlite:///:memory:")
|
||||||
|
self.tables = [
|
||||||
|
DatasourceRecord.__table__,
|
||||||
|
DatasourceGovernanceReferenceRecord.__table__,
|
||||||
|
DatasourcePayloadRecord.__table__,
|
||||||
|
DatasourcePayloadRowRecord.__table__,
|
||||||
|
DatasourceMaterializationRecord.__table__,
|
||||||
|
DatasourceStageRecord.__table__,
|
||||||
|
DatasourcePublicationRecord.__table__,
|
||||||
|
DatasourceLifecycleEvidenceRecord.__table__,
|
||||||
|
]
|
||||||
|
Base.metadata.create_all(self.engine, tables=self.tables)
|
||||||
|
self.session = sessionmaker(bind=self.engine)()
|
||||||
|
self.origins = _OriginProvider()
|
||||||
|
self.provider = SqlDatasourceProvider(registry=_Registry(self.origins))
|
||||||
|
self.writer = principal(
|
||||||
|
"writer",
|
||||||
|
scopes=(
|
||||||
|
CATALOGUE_READ_SCOPE,
|
||||||
|
SOURCE_WRITE_SCOPE,
|
||||||
|
STAGE_WRITE_SCOPE,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def tearDown(self) -> None:
|
||||||
|
self.session.close()
|
||||||
|
Base.metadata.drop_all(self.engine, tables=list(reversed(self.tables)))
|
||||||
|
self.engine.dispose()
|
||||||
|
|
||||||
|
def test_approval_quorum_is_attributable_idempotent_and_separated(self) -> None:
|
||||||
|
stage = self.provider.create_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage=DatasourceStageInput(
|
||||||
|
name="Approval register",
|
||||||
|
source_name="approval_register",
|
||||||
|
kind="upload",
|
||||||
|
mode="static",
|
||||||
|
shape="tabular",
|
||||||
|
rows=({"id": 1},),
|
||||||
|
governance=DatasourceGovernance(
|
||||||
|
approval_policy={
|
||||||
|
"version": "promotion-v2",
|
||||||
|
"required": True,
|
||||||
|
"required_approvals": 2,
|
||||||
|
"separation_of_duties": True,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self.assertEqual("awaiting_approval", stage.state)
|
||||||
|
policy_hash = str(stage.approval["policy_hash"])
|
||||||
|
subject_digest = str(stage.approval["subject_digest"])
|
||||||
|
|
||||||
|
creator_approver = principal(
|
||||||
|
"writer",
|
||||||
|
scopes=(STAGE_APPROVE_SCOPE,),
|
||||||
|
)
|
||||||
|
with self.assertRaisesRegex(
|
||||||
|
DatasourceValidationError,
|
||||||
|
"creator cannot approve",
|
||||||
|
):
|
||||||
|
self.provider.decide_stage(
|
||||||
|
self.session,
|
||||||
|
creator_approver,
|
||||||
|
stage_ref=stage.ref,
|
||||||
|
decision="approve",
|
||||||
|
reason="Creator review",
|
||||||
|
expected_policy_hash=policy_hash,
|
||||||
|
expected_subject_digest=subject_digest,
|
||||||
|
)
|
||||||
|
|
||||||
|
first, first_hash, replayed = self.provider.decide_stage(
|
||||||
|
self.session,
|
||||||
|
principal("approver-1", scopes=(STAGE_APPROVE_SCOPE,)),
|
||||||
|
stage_ref=stage.ref,
|
||||||
|
decision="approve",
|
||||||
|
reason="Quality and schema evidence reviewed.",
|
||||||
|
expected_policy_hash=policy_hash,
|
||||||
|
expected_subject_digest=subject_digest,
|
||||||
|
)
|
||||||
|
self.assertFalse(replayed)
|
||||||
|
self.assertTrue(first_hash)
|
||||||
|
self.assertEqual("awaiting_approval", first.state)
|
||||||
|
|
||||||
|
replay, replay_hash, replayed = self.provider.decide_stage(
|
||||||
|
self.session,
|
||||||
|
principal("approver-1", scopes=(STAGE_APPROVE_SCOPE,)),
|
||||||
|
stage_ref=stage.ref,
|
||||||
|
decision="approve",
|
||||||
|
reason="Quality and schema evidence reviewed.",
|
||||||
|
expected_policy_hash=policy_hash,
|
||||||
|
expected_subject_digest=subject_digest,
|
||||||
|
)
|
||||||
|
self.assertTrue(replayed)
|
||||||
|
self.assertIsNone(replay_hash)
|
||||||
|
self.assertEqual(1, replay.approval["approval_count"])
|
||||||
|
|
||||||
|
approved, second_hash, replayed = self.provider.decide_stage(
|
||||||
|
self.session,
|
||||||
|
principal("approver-2", scopes=(STAGE_APPROVE_SCOPE,)),
|
||||||
|
stage_ref=stage.ref,
|
||||||
|
decision="approve",
|
||||||
|
reason="Authority and promotion impact reviewed.",
|
||||||
|
expected_policy_hash=policy_hash,
|
||||||
|
expected_subject_digest=subject_digest,
|
||||||
|
)
|
||||||
|
self.assertFalse(replayed)
|
||||||
|
self.assertTrue(second_hash)
|
||||||
|
self.assertEqual("ready", approved.state)
|
||||||
|
|
||||||
|
datasource, materialization = self.provider.promote_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage_ref=stage.ref,
|
||||||
|
)
|
||||||
|
self.assertEqual(2, materialization.provenance["stage_approval"]["approval_count"])
|
||||||
|
self.assertTrue(materialization.provenance["promotion_evidence_hash"])
|
||||||
|
self.assertEqual(materialization.ref, datasource.current_materialization_ref)
|
||||||
|
|
||||||
|
evidence = tuple(
|
||||||
|
reversed(
|
||||||
|
self.provider.list_lifecycle_evidence(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
subject_ref=stage.ref,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
["stage.validated", "stage.approved", "stage.approved", "stage.promoted"],
|
||||||
|
[item.event_type for item in evidence],
|
||||||
|
)
|
||||||
|
for previous, current in zip(evidence, evidence[1:], strict=False):
|
||||||
|
self.assertEqual(previous.event_hash, current.previous_event_hash)
|
||||||
|
|
||||||
|
def test_approved_refresh_stage_is_required_before_current_changes(self) -> None:
|
||||||
|
datasource = self.provider.register_origin(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
origin_ref="origin:cases",
|
||||||
|
name="Cached cases",
|
||||||
|
source_name="cached_cases",
|
||||||
|
mode="cached",
|
||||||
|
governance=DatasourceGovernance(
|
||||||
|
approval_policy={
|
||||||
|
"version": "refresh-v1",
|
||||||
|
"required": True,
|
||||||
|
"required_approvals": 1,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
)
|
||||||
|
original_ref = datasource.current_materialization_ref
|
||||||
|
self.origins.rows = [{"id": 1, "status": "refreshed"}]
|
||||||
|
|
||||||
|
with self.assertRaisesRegex(
|
||||||
|
DatasourceValidationError,
|
||||||
|
"approved refresh stage",
|
||||||
|
):
|
||||||
|
self.provider.refresh_datasource(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
datasource_ref=datasource.ref,
|
||||||
|
)
|
||||||
|
|
||||||
|
stage = self.provider.prepare_refresh(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
datasource_ref=datasource.ref,
|
||||||
|
)
|
||||||
|
self.assertEqual("awaiting_approval", stage.state)
|
||||||
|
approved, _, _ = self.provider.decide_stage(
|
||||||
|
self.session,
|
||||||
|
principal("refresh-approver", scopes=(STAGE_APPROVE_SCOPE,)),
|
||||||
|
stage_ref=stage.ref,
|
||||||
|
decision="approve",
|
||||||
|
reason="The connector delta and schema are acceptable.",
|
||||||
|
expected_policy_hash=str(stage.approval["policy_hash"]),
|
||||||
|
expected_subject_digest=str(stage.approval["subject_digest"]),
|
||||||
|
)
|
||||||
|
self.assertEqual("ready", approved.state)
|
||||||
|
refreshed, current = self.provider.promote_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage_ref=stage.ref,
|
||||||
|
)
|
||||||
|
self.assertNotEqual(original_ref, refreshed.current_materialization_ref)
|
||||||
|
self.assertEqual(current.ref, refreshed.current_materialization_ref)
|
||||||
|
preview = self.provider.read_datasource(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
request=DatasourceReadRequest(datasource_ref=datasource.ref),
|
||||||
|
)
|
||||||
|
self.assertEqual("refreshed", preview.rows[0]["status"])
|
||||||
|
|
||||||
|
def test_retention_preview_blocks_current_and_held_evidence_then_purges_payload(self) -> None:
|
||||||
|
governance = DatasourceGovernance(
|
||||||
|
retention_policy={
|
||||||
|
"version": "records-v3",
|
||||||
|
"enabled": True,
|
||||||
|
"materialization_days": 1,
|
||||||
|
"frozen_evidence_days": 1,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
first_stage = self.provider.create_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage=DatasourceStageInput(
|
||||||
|
name="Retained register",
|
||||||
|
source_name="retained_register",
|
||||||
|
kind="upload",
|
||||||
|
mode="static",
|
||||||
|
shape="tabular",
|
||||||
|
rows=({"id": 1},),
|
||||||
|
csv_source=TabularCsvSource(text="id\n1\n"),
|
||||||
|
governance=governance,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
datasource, first = self.provider.promote_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage_ref=first_stage.ref,
|
||||||
|
)
|
||||||
|
second_stage = self.provider.create_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage=DatasourceStageInput(
|
||||||
|
name="Retained register",
|
||||||
|
source_name="retained_register",
|
||||||
|
kind="upload",
|
||||||
|
mode="static",
|
||||||
|
shape="tabular",
|
||||||
|
rows=({"id": 2},),
|
||||||
|
target_datasource_ref=datasource.ref,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
_, second = self.provider.promote_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage_ref=second_stage.ref,
|
||||||
|
)
|
||||||
|
as_of = max(first.created_at, second.created_at) + timedelta(days=2)
|
||||||
|
plan = self.provider.preview_retention(
|
||||||
|
self.session,
|
||||||
|
principal("admin", scopes=("datasources:source:admin",)),
|
||||||
|
as_of=as_of,
|
||||||
|
)
|
||||||
|
by_ref = {item.ref: item for item in plan.candidates}
|
||||||
|
self.assertTrue(by_ref[first.ref].eligible)
|
||||||
|
self.assertFalse(by_ref[second.ref].eligible)
|
||||||
|
self.assertIn("current_materialization", by_ref[second.ref].blockers)
|
||||||
|
|
||||||
|
first_row = self.session.scalar(
|
||||||
|
select(DatasourceMaterializationRecord).where(
|
||||||
|
DatasourceMaterializationRecord.id
|
||||||
|
== first.ref.removeprefix("materialization:")
|
||||||
|
)
|
||||||
|
)
|
||||||
|
payload_id = first_row.payload_id
|
||||||
|
self.assertIsNotNone(first_row.csv_source_)
|
||||||
|
disposed, evidence_hashes = self.provider.apply_retention(
|
||||||
|
self.session,
|
||||||
|
principal("admin", scopes=("datasources:source:admin",)),
|
||||||
|
as_of=as_of,
|
||||||
|
plan_hash=plan.plan_hash,
|
||||||
|
target_refs=(first.ref,),
|
||||||
|
)
|
||||||
|
self.assertEqual((first.ref,), disposed)
|
||||||
|
self.assertEqual(1, len(evidence_hashes))
|
||||||
|
self.assertIsNone(self.session.get(DatasourcePayloadRecord, payload_id))
|
||||||
|
self.assertIsNotNone(first_row.disposed_at)
|
||||||
|
self.assertIsNone(first_row.csv_source_)
|
||||||
|
self.assertEqual("disposed", first_row.state)
|
||||||
|
with self.assertRaises(DatasourceUnavailableError):
|
||||||
|
self.provider.read_datasource(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
request=DatasourceReadRequest(
|
||||||
|
datasource_ref=datasource.ref,
|
||||||
|
materialization_ref=first.ref,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
held = self.provider.update_datasource_governance(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
datasource_ref=datasource.ref,
|
||||||
|
governance=DatasourceGovernance(
|
||||||
|
retention_policy=governance.retention_policy,
|
||||||
|
hold_refs=("hold:legal-1",),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
frozen = self.provider.freeze_datasource(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
datasource_ref=held.ref,
|
||||||
|
label="Legal evidence",
|
||||||
|
)
|
||||||
|
held_plan = self.provider.preview_retention(
|
||||||
|
self.session,
|
||||||
|
principal("admin", scopes=("datasources:source:admin",)),
|
||||||
|
as_of=frozen.created_at + timedelta(days=2),
|
||||||
|
)
|
||||||
|
held_candidate = next(item for item in held_plan.candidates if item.ref == frozen.ref)
|
||||||
|
self.assertFalse(held_candidate.eligible)
|
||||||
|
self.assertIn("legal_hold", held_candidate.blockers)
|
||||||
|
|
||||||
|
def test_retention_deletes_only_an_explicitly_selected_eligible_stage(self) -> None:
|
||||||
|
stage = self.provider.create_stage(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
stage=DatasourceStageInput(
|
||||||
|
name="Transient import",
|
||||||
|
source_name="transient_import",
|
||||||
|
kind="upload",
|
||||||
|
mode="static",
|
||||||
|
shape="tabular",
|
||||||
|
rows=({"id": 1},),
|
||||||
|
csv_source=TabularCsvSource(text="id\n1\n"),
|
||||||
|
governance=DatasourceGovernance(
|
||||||
|
retention_policy={
|
||||||
|
"version": "stage-retention-v1",
|
||||||
|
"enabled": True,
|
||||||
|
"stage_days": 1,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
admin = principal(
|
||||||
|
"admin",
|
||||||
|
scopes=("datasources:source:admin",),
|
||||||
|
)
|
||||||
|
as_of = stage.created_at + timedelta(days=2)
|
||||||
|
plan = self.provider.preview_retention(
|
||||||
|
self.session,
|
||||||
|
admin,
|
||||||
|
as_of=as_of,
|
||||||
|
)
|
||||||
|
candidate = next(item for item in plan.candidates if item.ref == stage.ref)
|
||||||
|
self.assertTrue(candidate.eligible)
|
||||||
|
self.assertEqual("delete_stage", candidate.disposition)
|
||||||
|
|
||||||
|
disposed, evidence_hashes = self.provider.apply_retention(
|
||||||
|
self.session,
|
||||||
|
admin,
|
||||||
|
as_of=as_of,
|
||||||
|
plan_hash=plan.plan_hash,
|
||||||
|
target_refs=(stage.ref,),
|
||||||
|
)
|
||||||
|
self.assertEqual((stage.ref,), disposed)
|
||||||
|
self.assertEqual(1, len(evidence_hashes))
|
||||||
|
self.assertIsNone(
|
||||||
|
self.session.get(
|
||||||
|
DatasourceStageRecord,
|
||||||
|
stage.ref.removeprefix("stage:"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
evidence = self.provider.list_lifecycle_evidence(
|
||||||
|
self.session,
|
||||||
|
self.writer,
|
||||||
|
subject_ref=stage.ref,
|
||||||
|
)
|
||||||
|
self.assertEqual("retention.stage_deleted", evidence[0].event_type)
|
||||||
|
self.assertEqual(plan.plan_hash, evidence[0].details_["plan_hash"])
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
PAGE = ROOT / "webui" / "src" / "features" / "datasources" / "DatasourcesPage.tsx"
|
||||||
|
API = ROOT / "webui" / "src" / "api" / "datasources.ts"
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceLifecycleGovernanceUiTests(unittest.TestCase):
|
||||||
|
def test_stage_decisions_and_retention_use_shared_semantic_primitives(self) -> None:
|
||||||
|
source = PAGE.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
self.assertIn("<WorkspaceActionBar", source)
|
||||||
|
self.assertIn("function StageDecisionDialog", source)
|
||||||
|
self.assertIn("function RetentionDialog", source)
|
||||||
|
self.assertIn("<ConfirmDialog", source)
|
||||||
|
self.assertIn('variant="danger"', source)
|
||||||
|
self.assertIn("datasources.field.approval-policy", source)
|
||||||
|
self.assertIn("datasources.field.retention-policy-contract", source)
|
||||||
|
self.assertIn('state === "awaiting_approval"', source)
|
||||||
|
|
||||||
|
def test_client_binds_decisions_and_apply_to_server_evidence(self) -> None:
|
||||||
|
source = API.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
self.assertIn("expected_policy_hash", source)
|
||||||
|
self.assertIn("expected_subject_digest", source)
|
||||||
|
self.assertIn("/lifecycle-evidence", source)
|
||||||
|
self.assertIn("/retention/plan", source)
|
||||||
|
self.assertIn("plan_hash: plan.plan_hash", source)
|
||||||
|
self.assertIn("/refresh/stage", source)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -31,6 +31,10 @@ class DatasourceManifestTests(unittest.TestCase):
|
|||||||
if item.name == "connectors.datasource_origins"
|
if item.name == "connectors.datasource_origins"
|
||||||
)
|
)
|
||||||
self.assertTrue(requirement.optional)
|
self.assertTrue(requirement.optional)
|
||||||
|
self.assertIn(
|
||||||
|
"datasources:stage:approve",
|
||||||
|
{item.scope for item in manifest.permissions},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -14,6 +14,47 @@ from govoplan_datasources.backend.manifest import get_manifest
|
|||||||
|
|
||||||
|
|
||||||
class DatasourceMigrationTests(unittest.TestCase):
|
class DatasourceMigrationTests(unittest.TestCase):
|
||||||
|
def test_csv_evidence_upgrade_preserves_legacy_rows_and_fingerprints(self) -> None:
|
||||||
|
with tempfile.TemporaryDirectory(prefix="govoplan-datasources-csv-migration-") as directory:
|
||||||
|
url = f"sqlite:///{Path(directory) / 'datasources.db'}"
|
||||||
|
config = self._config(url)
|
||||||
|
command.upgrade(config, "d1a7c3e9f5b2")
|
||||||
|
engine = create_engine(url)
|
||||||
|
try:
|
||||||
|
metadata = MetaData()
|
||||||
|
tables = {name: Table(name, metadata, autoload_with=engine) for name in (
|
||||||
|
"datasource_catalogue", "datasource_stages", "datasource_materializations",
|
||||||
|
)}
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
common = dict(tenant_id="tenant-1", schema=[], fingerprint="a" * 64,
|
||||||
|
metadata={"retained": True}, provenance={}, created_at=now, updated_at=now)
|
||||||
|
with engine.begin() as connection:
|
||||||
|
connection.execute(tables["datasource_catalogue"].insert().values(
|
||||||
|
**common, id="legacy-source", source_name="legacy", name="Legacy", kind="upload",
|
||||||
|
mode="static", shape="tabular", status="active", schema_version=1,
|
||||||
|
))
|
||||||
|
connection.execute(tables["datasource_stages"].insert().values(
|
||||||
|
**common, id="legacy-stage", source_name="staged", name="Staged", kind="upload",
|
||||||
|
mode="static", shape="tabular", state="ready", rows=[{"id": 1}],
|
||||||
|
row_count=1, byte_count=10, validation={}, governance={}, approval={},
|
||||||
|
))
|
||||||
|
connection.execute(tables["datasource_materializations"].insert().values(
|
||||||
|
**common, id="legacy-materialization", datasource_id="legacy-source", revision=1,
|
||||||
|
state="published", schema_version=1, rows=[{"id": 1}], row_count=1,
|
||||||
|
byte_count=10, governance_snapshot={},
|
||||||
|
))
|
||||||
|
before = {name: dict(connection.execute(select(table)).mappings().one()) for name, table in tables.items()}
|
||||||
|
command.upgrade(config, "e2b8d4a0f6c3")
|
||||||
|
with engine.connect() as connection:
|
||||||
|
for name in tables:
|
||||||
|
upgraded = Table(name, MetaData(), autoload_with=connection)
|
||||||
|
after = dict(connection.execute(select(upgraded)).mappings().one())
|
||||||
|
if name != "datasource_catalogue":
|
||||||
|
self.assertIsNone(after.pop("csv_source"))
|
||||||
|
self.assertEqual(before[name], after)
|
||||||
|
finally:
|
||||||
|
engine.dispose()
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _config(url: str):
|
def _config(url: str):
|
||||||
return alembic_config(
|
return alembic_config(
|
||||||
@@ -34,7 +75,7 @@ class DatasourceMigrationTests(unittest.TestCase):
|
|||||||
try:
|
try:
|
||||||
with engine.connect() as connection:
|
with engine.connect() as connection:
|
||||||
self.assertIn(
|
self.assertIn(
|
||||||
"b8d2f5a0c3e7",
|
"e2b8d4a0f6c3",
|
||||||
set(MigrationContext.configure(connection).get_current_heads()),
|
set(MigrationContext.configure(connection).get_current_heads()),
|
||||||
)
|
)
|
||||||
catalogue_columns = {
|
catalogue_columns = {
|
||||||
@@ -50,13 +91,36 @@ class DatasourceMigrationTests(unittest.TestCase):
|
|||||||
"publication_state",
|
"publication_state",
|
||||||
"owner_ref",
|
"owner_ref",
|
||||||
"quality_policy",
|
"quality_policy",
|
||||||
|
"access_policy_ref",
|
||||||
|
"visibility_policy",
|
||||||
|
"approval_policy",
|
||||||
|
"retention_policy",
|
||||||
"dependency_refs",
|
"dependency_refs",
|
||||||
}.issubset(catalogue_columns)
|
}.issubset(catalogue_columns)
|
||||||
)
|
)
|
||||||
|
stage_columns = {
|
||||||
|
item["name"]
|
||||||
|
for item in inspect(connection).get_columns(
|
||||||
|
"datasource_stages"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
materialization_columns = {
|
||||||
|
item["name"]
|
||||||
|
for item in inspect(connection).get_columns(
|
||||||
|
"datasource_materializations"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
self.assertIn("approval", stage_columns)
|
||||||
|
self.assertTrue(
|
||||||
|
{"disposed_at", "disposition"}.issubset(
|
||||||
|
materialization_columns
|
||||||
|
)
|
||||||
|
)
|
||||||
self.assertEqual(
|
self.assertEqual(
|
||||||
{
|
{
|
||||||
"datasource_catalogue",
|
"datasource_catalogue",
|
||||||
"datasource_governance_references",
|
"datasource_governance_references",
|
||||||
|
"datasource_lifecycle_evidence",
|
||||||
"datasource_materializations",
|
"datasource_materializations",
|
||||||
"datasource_payload_rows",
|
"datasource_payload_rows",
|
||||||
"datasource_payloads",
|
"datasource_payloads",
|
||||||
|
|||||||
@@ -0,0 +1,468 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from sqlalchemy import create_engine
|
||||||
|
from sqlalchemy.orm import sessionmaker
|
||||||
|
|
||||||
|
from govoplan_core.auth import ApiPrincipal
|
||||||
|
from govoplan_core.core.access import PrincipalRef
|
||||||
|
from govoplan_core.core.change_sequence import ChangeSequenceEntry
|
||||||
|
from govoplan_core.core.datasources import (
|
||||||
|
CAPABILITY_DATASOURCE_ORIGINS,
|
||||||
|
CAPABILITY_POLICY_DATASOURCE_VISIBILITY,
|
||||||
|
DatasourceAccessError,
|
||||||
|
DatasourceGovernance,
|
||||||
|
DatasourceNotFoundError,
|
||||||
|
DatasourceField,
|
||||||
|
DatasourceOrigin,
|
||||||
|
DatasourceOriginReadResult,
|
||||||
|
DatasourceReadRequest,
|
||||||
|
DatasourceVisibilityPolicyDecision,
|
||||||
|
)
|
||||||
|
from govoplan_core.db.base import Base, utcnow
|
||||||
|
from govoplan_datasources.backend.db.models import (
|
||||||
|
DatasourceGovernanceReferenceRecord,
|
||||||
|
DatasourceMaterializationRecord,
|
||||||
|
DatasourcePayloadRecord,
|
||||||
|
DatasourceRecord,
|
||||||
|
)
|
||||||
|
from govoplan_datasources.backend.service import (
|
||||||
|
CATALOGUE_READ_SCOPE,
|
||||||
|
SqlDatasourceProvider,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
SCHEMA = [
|
||||||
|
{"name": "id", "data_type": "integer", "nullable": False},
|
||||||
|
{"name": "owner_id", "data_type": "string", "nullable": False},
|
||||||
|
{"name": "secret", "data_type": "string", "nullable": False},
|
||||||
|
{"name": "internal_note", "data_type": "string", "nullable": True},
|
||||||
|
]
|
||||||
|
ROWS = [
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"owner_id": "account-1",
|
||||||
|
"secret": "protected-one",
|
||||||
|
"internal_note": "hidden-one",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 2,
|
||||||
|
"owner_id": "account-2",
|
||||||
|
"secret": "protected-two",
|
||||||
|
"internal_note": "hidden-two",
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def principal(
|
||||||
|
*,
|
||||||
|
tenant_id: str = "tenant-1",
|
||||||
|
account_id: str = "account-1",
|
||||||
|
role_ids: tuple[str, ...] = ("reader",),
|
||||||
|
auth_method: str = "session",
|
||||||
|
service_account_id: str | None = None,
|
||||||
|
) -> ApiPrincipal:
|
||||||
|
return ApiPrincipal(
|
||||||
|
principal=PrincipalRef(
|
||||||
|
account_id=account_id,
|
||||||
|
membership_id=(None if auth_method == "service_account" else "member-1"),
|
||||||
|
tenant_id=tenant_id,
|
||||||
|
scopes=frozenset({CATALOGUE_READ_SCOPE}),
|
||||||
|
role_ids=frozenset(role_ids),
|
||||||
|
auth_method=auth_method,
|
||||||
|
service_account_id=service_account_id,
|
||||||
|
),
|
||||||
|
account=object(),
|
||||||
|
user=object(),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class _PolicyProvider:
|
||||||
|
def __init__(self, decision: DatasourceVisibilityPolicyDecision) -> None:
|
||||||
|
self.decision = decision
|
||||||
|
self.requests = []
|
||||||
|
|
||||||
|
def decide_datasource_visibility(self, _session, *, request):
|
||||||
|
self.requests.append(request)
|
||||||
|
return self.decision
|
||||||
|
|
||||||
|
|
||||||
|
class _Registry:
|
||||||
|
def __init__(self, policy_provider: _PolicyProvider) -> None:
|
||||||
|
self.policy_provider = policy_provider
|
||||||
|
|
||||||
|
def has_capability(self, name: str) -> bool:
|
||||||
|
return name == CAPABILITY_POLICY_DATASOURCE_VISIBILITY
|
||||||
|
|
||||||
|
def capability(self, name: str) -> object:
|
||||||
|
if not self.has_capability(name):
|
||||||
|
raise KeyError(name)
|
||||||
|
return self.policy_provider
|
||||||
|
|
||||||
|
|
||||||
|
class _OriginProvider:
|
||||||
|
origin = DatasourceOrigin(
|
||||||
|
ref="secret:origin",
|
||||||
|
source_name="live_cases",
|
||||||
|
name="Live cases",
|
||||||
|
kind="database",
|
||||||
|
shape="tabular",
|
||||||
|
supported_modes=("live",),
|
||||||
|
provider="test",
|
||||||
|
schema=tuple(
|
||||||
|
DatasourceField(
|
||||||
|
name=str(field["name"]),
|
||||||
|
data_type=str(field["data_type"]),
|
||||||
|
nullable=bool(field["nullable"]),
|
||||||
|
)
|
||||||
|
for field in SCHEMA
|
||||||
|
),
|
||||||
|
fingerprint="live-base-fingerprint",
|
||||||
|
row_count=2,
|
||||||
|
)
|
||||||
|
|
||||||
|
def list_origins(self, _session, _principal, *, query="", limit=100):
|
||||||
|
del query
|
||||||
|
return (self.origin,)[:limit]
|
||||||
|
|
||||||
|
def get_origin(self, _session, _principal, *, origin_ref):
|
||||||
|
return self.origin if origin_ref == self.origin.ref else None
|
||||||
|
|
||||||
|
def read_origin(self, _session, _principal, *, request):
|
||||||
|
rows = ROWS[request.offset : request.offset + request.limit]
|
||||||
|
return DatasourceOriginReadResult(
|
||||||
|
origin=self.origin,
|
||||||
|
rows=tuple(rows),
|
||||||
|
total_rows=2,
|
||||||
|
truncated=request.offset + len(rows) < 2,
|
||||||
|
elapsed_ms=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class _LiveRegistry:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.origin_provider = _OriginProvider()
|
||||||
|
|
||||||
|
def has_capability(self, name: str) -> bool:
|
||||||
|
return name == CAPABILITY_DATASOURCE_ORIGINS
|
||||||
|
|
||||||
|
def capability(self, name: str) -> object:
|
||||||
|
if not self.has_capability(name):
|
||||||
|
raise KeyError(name)
|
||||||
|
return self.origin_provider
|
||||||
|
|
||||||
|
|
||||||
|
class DatasourceVisibilityTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.engine = create_engine("sqlite:///:memory:")
|
||||||
|
Base.metadata.create_all(
|
||||||
|
self.engine,
|
||||||
|
tables=[
|
||||||
|
DatasourceRecord.__table__,
|
||||||
|
DatasourceGovernanceReferenceRecord.__table__,
|
||||||
|
DatasourcePayloadRecord.__table__,
|
||||||
|
DatasourceMaterializationRecord.__table__,
|
||||||
|
ChangeSequenceEntry.__table__,
|
||||||
|
],
|
||||||
|
)
|
||||||
|
self.Session = sessionmaker(bind=self.engine)
|
||||||
|
self.session = self.Session()
|
||||||
|
|
||||||
|
def tearDown(self) -> None:
|
||||||
|
self.session.close()
|
||||||
|
Base.metadata.drop_all(
|
||||||
|
self.engine,
|
||||||
|
tables=[
|
||||||
|
DatasourceMaterializationRecord.__table__,
|
||||||
|
ChangeSequenceEntry.__table__,
|
||||||
|
DatasourcePayloadRecord.__table__,
|
||||||
|
DatasourceGovernanceReferenceRecord.__table__,
|
||||||
|
DatasourceRecord.__table__,
|
||||||
|
],
|
||||||
|
)
|
||||||
|
self.engine.dispose()
|
||||||
|
|
||||||
|
def _datasource(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
policy: dict[str, object] | None = None,
|
||||||
|
access_policy_ref: str | None = None,
|
||||||
|
frozen: bool = False,
|
||||||
|
snapshot_policy: dict[str, object] | None = None,
|
||||||
|
) -> DatasourceRecord:
|
||||||
|
item = DatasourceRecord(
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
source_name="governed_cases",
|
||||||
|
name="Governed cases",
|
||||||
|
kind="upload",
|
||||||
|
mode="static",
|
||||||
|
shape="tabular",
|
||||||
|
status="active",
|
||||||
|
provider="private.provider",
|
||||||
|
provider_ref="secret:origin",
|
||||||
|
schema_=SCHEMA,
|
||||||
|
fingerprint="base-fingerprint",
|
||||||
|
row_count=2,
|
||||||
|
byte_count=250,
|
||||||
|
provenance_={"secret_locator": "private://rows"},
|
||||||
|
metadata_={"credential_ref": "credential:secret"},
|
||||||
|
access_policy_ref=access_policy_ref,
|
||||||
|
visibility_policy=policy or {},
|
||||||
|
)
|
||||||
|
self.session.add(item)
|
||||||
|
self.session.flush()
|
||||||
|
snapshot = DatasourceGovernance(
|
||||||
|
publication_state="internal",
|
||||||
|
visibility_policy=snapshot_policy or policy or {},
|
||||||
|
access_policy_ref=access_policy_ref,
|
||||||
|
)
|
||||||
|
materialization = DatasourceMaterializationRecord(
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
datasource_id=item.id,
|
||||||
|
revision=1,
|
||||||
|
state="published",
|
||||||
|
schema_=SCHEMA,
|
||||||
|
rows=ROWS,
|
||||||
|
fingerprint="materialization-fingerprint",
|
||||||
|
row_count=2,
|
||||||
|
byte_count=250,
|
||||||
|
frozen_at=utcnow() if frozen else None,
|
||||||
|
frozen_label="Evidence" if frozen else None,
|
||||||
|
provenance_={"protected": "source details"},
|
||||||
|
metadata_={"protected": "materialization details"},
|
||||||
|
governance_snapshot_=snapshot.to_dict(),
|
||||||
|
)
|
||||||
|
self.session.add(materialization)
|
||||||
|
self.session.flush()
|
||||||
|
item.current_materialization_id = materialization.id
|
||||||
|
self.session.flush()
|
||||||
|
return item
|
||||||
|
|
||||||
|
def test_role_acl_filters_discovery_and_denies_reads(self) -> None:
|
||||||
|
item = self._datasource(policy={"source_acl": {"role_ids": ["reader"]}})
|
||||||
|
provider = SqlDatasourceProvider()
|
||||||
|
|
||||||
|
self.assertEqual(1, len(provider.list_datasources(self.session, principal())))
|
||||||
|
denied = principal(role_ids=("other",))
|
||||||
|
self.assertEqual((), provider.list_datasources(self.session, denied))
|
||||||
|
self.assertIsNone(
|
||||||
|
provider.get_datasource(
|
||||||
|
self.session,
|
||||||
|
denied,
|
||||||
|
datasource_ref=f"datasource:{item.id}",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
with self.assertRaises(DatasourceAccessError):
|
||||||
|
provider.read_datasource(
|
||||||
|
self.session,
|
||||||
|
denied,
|
||||||
|
request=DatasourceReadRequest(datasource_ref=f"datasource:{item.id}"),
|
||||||
|
)
|
||||||
|
other_tenant = principal(tenant_id="tenant-2")
|
||||||
|
self.assertEqual((), provider.list_datasources(self.session, other_tenant))
|
||||||
|
with self.assertRaises(DatasourceNotFoundError):
|
||||||
|
provider.read_datasource(
|
||||||
|
self.session,
|
||||||
|
other_tenant,
|
||||||
|
request=DatasourceReadRequest(datasource_ref=f"datasource:{item.id}"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_rows_and_fields_are_filtered_into_an_opaque_permitted_view(self) -> None:
|
||||||
|
policy = {
|
||||||
|
"source_acl": {"role_ids": ["reader"]},
|
||||||
|
"fields": {
|
||||||
|
"secret": {
|
||||||
|
"classification": "restricted",
|
||||||
|
"action": "redact",
|
||||||
|
"allow": {"role_ids": ["privileged"]},
|
||||||
|
},
|
||||||
|
"internal_note": {
|
||||||
|
"classification": "confidential",
|
||||||
|
"action": "omit",
|
||||||
|
"allow": {"role_ids": ["privileged"]},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"row_filters": [
|
||||||
|
{"field": "owner_id", "claim": "account_id", "operator": "equals"}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
item = self._datasource(policy=policy)
|
||||||
|
provider = SqlDatasourceProvider()
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"govoplan_datasources.backend.service.audit_event"
|
||||||
|
) as audit_event:
|
||||||
|
result = provider.read_datasource(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
request=DatasourceReadRequest(
|
||||||
|
datasource_ref=f"datasource:{item.id}"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(
|
||||||
|
({"id": 1, "owner_id": "account-1", "secret": None},),
|
||||||
|
result.rows,
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
["id", "owner_id", "secret"],
|
||||||
|
[field.name for field in result.datasource.schema],
|
||||||
|
)
|
||||||
|
self.assertEqual("restricted", result.datasource.schema[-1].classification)
|
||||||
|
self.assertNotEqual(
|
||||||
|
"materialization-fingerprint", result.datasource.fingerprint
|
||||||
|
)
|
||||||
|
self.assertIsNone(result.datasource.provider)
|
||||||
|
self.assertIsNone(result.datasource.provider_ref)
|
||||||
|
self.assertNotIn("credential_ref", result.datasource.metadata)
|
||||||
|
self.assertEqual(1, result.total_rows)
|
||||||
|
self.assertEqual(
|
||||||
|
["datasource.visibility_applied"],
|
||||||
|
[diagnostic.code for diagnostic in result.diagnostics],
|
||||||
|
)
|
||||||
|
audit_details = audit_event.call_args.kwargs["details"]
|
||||||
|
self.assertEqual(2, audit_details["visibility_summary"]["field_rule_count"])
|
||||||
|
self.assertNotIn("protected-one", str(audit_details))
|
||||||
|
|
||||||
|
repeated = provider.read_datasource(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
request=DatasourceReadRequest(
|
||||||
|
datasource_ref=f"datasource:{item.id}",
|
||||||
|
expected_fingerprint=result.datasource.fingerprint,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self.assertEqual(result.datasource.fingerprint, repeated.datasource.fingerprint)
|
||||||
|
|
||||||
|
def test_materialization_acl_supports_service_principals(self) -> None:
|
||||||
|
item = self._datasource(
|
||||||
|
policy={
|
||||||
|
"source_acl": {"auth_methods": ["session", "service_account"]},
|
||||||
|
"materialization_acl": {"service_account_ids": ["service:reporting"]},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
provider = SqlDatasourceProvider()
|
||||||
|
|
||||||
|
with self.assertRaises(DatasourceAccessError):
|
||||||
|
provider.read_datasource(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
request=DatasourceReadRequest(datasource_ref=f"datasource:{item.id}"),
|
||||||
|
)
|
||||||
|
result = provider.read_datasource(
|
||||||
|
self.session,
|
||||||
|
principal(
|
||||||
|
account_id="service-account",
|
||||||
|
auth_method="service_account",
|
||||||
|
service_account_id="service:reporting",
|
||||||
|
),
|
||||||
|
request=DatasourceReadRequest(datasource_ref=f"datasource:{item.id}"),
|
||||||
|
)
|
||||||
|
self.assertEqual(2, result.total_rows)
|
||||||
|
|
||||||
|
def test_frozen_state_keeps_its_restrictive_local_policy(self) -> None:
|
||||||
|
snapshot_policy = {"source_acl": {"role_ids": ["evidence-reader"]}}
|
||||||
|
item = self._datasource(
|
||||||
|
policy={},
|
||||||
|
frozen=True,
|
||||||
|
snapshot_policy=snapshot_policy,
|
||||||
|
)
|
||||||
|
provider = SqlDatasourceProvider()
|
||||||
|
|
||||||
|
with self.assertRaises(DatasourceAccessError):
|
||||||
|
provider.read_datasource(
|
||||||
|
self.session,
|
||||||
|
principal(role_ids=("reader",)),
|
||||||
|
request=DatasourceReadRequest(
|
||||||
|
datasource_ref=f"datasource:{item.id}",
|
||||||
|
consistency="frozen",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
result = provider.read_datasource(
|
||||||
|
self.session,
|
||||||
|
principal(role_ids=("evidence-reader",)),
|
||||||
|
request=DatasourceReadRequest(
|
||||||
|
datasource_ref=f"datasource:{item.id}",
|
||||||
|
consistency="frozen",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self.assertEqual(2, result.total_rows)
|
||||||
|
|
||||||
|
def test_external_policy_tightens_local_behavior_and_missing_provider_denies(
|
||||||
|
self,
|
||||||
|
) -> None:
|
||||||
|
item = self._datasource(access_policy_ref="case-workers")
|
||||||
|
with self.assertRaises(DatasourceAccessError):
|
||||||
|
SqlDatasourceProvider().read_datasource(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
request=DatasourceReadRequest(datasource_ref=f"datasource:{item.id}"),
|
||||||
|
)
|
||||||
|
|
||||||
|
external = _PolicyProvider(
|
||||||
|
DatasourceVisibilityPolicyDecision(
|
||||||
|
allowed=True,
|
||||||
|
policies=({"source_acl": {"role_ids": ["case-worker"]}},),
|
||||||
|
decision_ref="policy-decision:1",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
provider = SqlDatasourceProvider(registry=_Registry(external))
|
||||||
|
result = provider.read_datasource(
|
||||||
|
self.session,
|
||||||
|
principal(role_ids=("case-worker",)),
|
||||||
|
request=DatasourceReadRequest(datasource_ref=f"datasource:{item.id}"),
|
||||||
|
)
|
||||||
|
self.assertEqual(2, result.total_rows)
|
||||||
|
self.assertEqual("case-workers", external.requests[-1].policy_ref)
|
||||||
|
|
||||||
|
def test_denied_audit_contains_no_protected_values(self) -> None:
|
||||||
|
item = self._datasource(policy={"source_acl": {"role_ids": ["authorized"]}})
|
||||||
|
provider = SqlDatasourceProvider()
|
||||||
|
with patch("govoplan_datasources.backend.service.audit_event") as audit_event:
|
||||||
|
with self.assertRaises(DatasourceAccessError):
|
||||||
|
provider.read_datasource(
|
||||||
|
self.session,
|
||||||
|
principal(role_ids=("denied",)),
|
||||||
|
request=DatasourceReadRequest(
|
||||||
|
datasource_ref=f"datasource:{item.id}"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
details = audit_event.call_args.kwargs["details"]
|
||||||
|
self.assertEqual("denied", details["outcome"])
|
||||||
|
serialized = str(details)
|
||||||
|
self.assertNotIn("protected-one", serialized)
|
||||||
|
self.assertNotIn("hidden-one", serialized)
|
||||||
|
self.assertNotIn("credential:secret", serialized)
|
||||||
|
|
||||||
|
def test_live_rows_are_filtered_before_origin_data_is_returned(self) -> None:
|
||||||
|
item = self._datasource(
|
||||||
|
policy={
|
||||||
|
"source_acl": {"role_ids": ["reader"]},
|
||||||
|
"row_filters": [
|
||||||
|
{"field": "owner_id", "claim": "account_id"}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
item.mode = "live"
|
||||||
|
self.session.flush()
|
||||||
|
|
||||||
|
result = SqlDatasourceProvider(registry=_LiveRegistry()).read_datasource(
|
||||||
|
self.session,
|
||||||
|
principal(),
|
||||||
|
request=DatasourceReadRequest(
|
||||||
|
datasource_ref=f"datasource:{item.id}",
|
||||||
|
consistency="live",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(1, result.total_rows)
|
||||||
|
self.assertEqual("account-1", result.rows[0]["owner_id"])
|
||||||
|
self.assertNotEqual("live-base-fingerprint", result.datasource.fingerprint)
|
||||||
|
self.assertIsNone(result.datasource.provider_ref)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
+4
-3
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@govoplan/datasources-webui",
|
"name": "@govoplan/datasources-webui",
|
||||||
"version": "0.1.19",
|
"version": "0.1.26",
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "src/index.ts",
|
"main": "src/index.ts",
|
||||||
@@ -14,10 +14,11 @@
|
|||||||
"./styles/datasources.css": "./src/styles/datasources.css"
|
"./styles/datasources.css": "./src/styles/datasources.css"
|
||||||
},
|
},
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"typecheck": "tsc --noEmit"
|
"typecheck": "tsc --noEmit",
|
||||||
|
"test:detail-refresh": "node --test scripts/test-detail-refresh.mjs"
|
||||||
},
|
},
|
||||||
"peerDependencies": {
|
"peerDependencies": {
|
||||||
"@govoplan/core-webui": "^0.1.18",
|
"@govoplan/core-webui": "^0.1.46",
|
||||||
"lucide-react": "^1.23.0",
|
"lucide-react": "^1.23.0",
|
||||||
"react": ">=19.2.7 <20",
|
"react": ">=19.2.7 <20",
|
||||||
"react-dom": ">=19.2.7 <20",
|
"react-dom": ">=19.2.7 <20",
|
||||||
|
|||||||
Executable
+258
@@ -0,0 +1,258 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { readFileSync } from "node:fs";
|
||||||
|
import { createRequire } from "node:module";
|
||||||
|
import test from "node:test";
|
||||||
|
|
||||||
|
const require = createRequire(new URL("../../../govoplan-core/webui/package.json", import.meta.url));
|
||||||
|
const { transformSync } = require("esbuild");
|
||||||
|
const page = readFileSync(new URL("../src/features/datasources/DatasourcesPage.tsx", import.meta.url), "utf8");
|
||||||
|
function extract(startMarker, endMarker) {
|
||||||
|
const start = page.indexOf(startMarker);
|
||||||
|
const end = page.indexOf(endMarker, start);
|
||||||
|
assert.ok(start >= 0 && end > start, "test the actual page closures");
|
||||||
|
return page.slice(start + startMarker.length, end);
|
||||||
|
}
|
||||||
|
const reloadBody = extract(" const reload = useCallback(async (preferredDatasourceRef?: string) => {", " }, [dialogScope]);");
|
||||||
|
const detailBody = extract(" useEffect(() => {\n if (!selectedDatasourceRef || loading || !catalogueReady)", " }, [selectedDatasourceRef, detailScope, loading, catalogueReady]);");
|
||||||
|
function evaluate(body, bindings) {
|
||||||
|
const code = transformSync(body, { loader: "ts", target: "es2022" }).code;
|
||||||
|
return new Function(...Object.keys(bindings), code)(...Object.values(bindings));
|
||||||
|
}
|
||||||
|
const deferred = () => {
|
||||||
|
let resolve, reject;
|
||||||
|
const promise = new Promise((yes, no) => { resolve = yes; reject = no; });
|
||||||
|
return { promise, resolve, reject };
|
||||||
|
};
|
||||||
|
const settle = async () => { await Promise.resolve(); await Promise.resolve(); await Promise.resolve(); };
|
||||||
|
function harness() {
|
||||||
|
const state = { selectedDatasourceRef: "source-1", loading: false, detailRevision: 0, preview: null, materializations: [], detailLoading: false, catalogueAuthority: 0 };
|
||||||
|
const settings = {};
|
||||||
|
const auth = {};
|
||||||
|
const bindings = {
|
||||||
|
settings, auth, reloadRequestId: { current: 0 }, authority: { current: "authority-A" }, authorityKey: "authority-A", dialogScope: 0, authorityEpoch: { current: { revision: 0 } },
|
||||||
|
currentDetailScope: { current: null }, apiErrorMessage: String,
|
||||||
|
listDatasources: async () => [{ ref: "source-1" }], listDatasourceStages: async () => [],
|
||||||
|
listDatasourceOrigins: async () => ({ origins: [], available: false })
|
||||||
|
};
|
||||||
|
bindings.isCurrentAuthority = () => bindings.authority.current === "authority-A" && bindings.authorityEpoch.current.revision === 0;
|
||||||
|
for (const key of ["loading", "error", "datasources", "stages", "origins", "originsAvailable", "detailRevision", "selectedDatasourceRef", "selectedStageRef", "selectedOriginRef", "preview", "materializations", "detailLoading", "detailResponseScope", "catalogueAuthority", "working", "success", "view", "addOpen", "freezeOpen", "freezeLabel", "retireOpen", "promoteOpen", "governanceOpen", "decisionOpen", "retentionOpen"]) {
|
||||||
|
bindings[`set${key[0].toUpperCase()}${key.slice(1)}`] = (value) => { state[key] = typeof value === "function" ? value(state[key]) : value; };
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
state, bindings,
|
||||||
|
reload: (preferredDatasourceRef) => evaluate(`return (async () => {${reloadBody}})();`, { ...bindings, preferredDatasourceRef }),
|
||||||
|
startDetail: () => {
|
||||||
|
const preview = deferred();
|
||||||
|
const history = deferred();
|
||||||
|
const scope = {};
|
||||||
|
bindings.currentDetailScope.current = scope;
|
||||||
|
const cleanup = evaluate(`if (!selectedDatasourceRef || loading || !catalogueReady)${detailBody}`, {
|
||||||
|
...bindings, selectedDatasourceRef: state.selectedDatasourceRef, loading: state.loading, detailScope: scope,
|
||||||
|
catalogueReady: state.catalogueAuthority === bindings.authorityEpoch.current.revision,
|
||||||
|
previewDatasource: () => preview.promise, listDatasourceMaterializations: () => history.promise
|
||||||
|
});
|
||||||
|
return { preview, history, scope, cleanup };
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
test("same-reference reload invalidates preview/history once, including freeze with an unchanged current materialization", async () => {
|
||||||
|
const h = harness();
|
||||||
|
for (const operation of ["manual reload", "refresh", "freeze", "promotion"]) {
|
||||||
|
const previous = h.state.detailRevision;
|
||||||
|
await h.reload("source-1");
|
||||||
|
assert.equal(h.state.selectedDatasourceRef, "source-1", operation);
|
||||||
|
assert.equal(h.state.detailRevision, previous + 1, operation);
|
||||||
|
const detail = h.startDetail();
|
||||||
|
detail.preview.resolve({ rows: [{ value: operation }] });
|
||||||
|
detail.history.resolve([{ revision: h.state.detailRevision }]);
|
||||||
|
await settle();
|
||||||
|
assert.equal(h.state.preview.rows[0].value, operation);
|
||||||
|
assert.equal(h.state.materializations[0].revision, h.state.detailRevision);
|
||||||
|
assert.equal(h.state.detailResponseScope, detail.scope);
|
||||||
|
detail.cleanup();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a changed scope immediately suppresses old preview, history and errors, even before effect cleanup", async () => {
|
||||||
|
const h = harness();
|
||||||
|
const old = h.startDetail();
|
||||||
|
const current = h.startDetail();
|
||||||
|
current.preview.resolve({ rows: [{ value: "current" }] });
|
||||||
|
current.history.resolve([{ revision: 2 }]);
|
||||||
|
await settle();
|
||||||
|
old.preview.reject(new Error("stale error"));
|
||||||
|
old.history.resolve([{ revision: 1 }]);
|
||||||
|
await settle();
|
||||||
|
assert.equal(h.state.preview.rows[0].value, "current");
|
||||||
|
assert.equal(h.state.materializations[0].revision, 2);
|
||||||
|
assert.equal(h.state.error, undefined);
|
||||||
|
current.cleanup();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("unmount cancellation and partial detail failures preserve allSettled behavior", async () => {
|
||||||
|
const h = harness();
|
||||||
|
const cancelled = h.startDetail();
|
||||||
|
cancelled.cleanup();
|
||||||
|
cancelled.preview.resolve({ rows: ["obsolete"] });
|
||||||
|
cancelled.history.resolve([]);
|
||||||
|
await settle();
|
||||||
|
assert.equal(h.state.preview, null);
|
||||||
|
const current = h.startDetail();
|
||||||
|
current.preview.reject(new Error("current preview denied"));
|
||||||
|
current.history.resolve([{ revision: 3 }]);
|
||||||
|
await settle();
|
||||||
|
assert.equal(h.state.preview, null);
|
||||||
|
assert.equal(h.state.materializations[0].revision, 3);
|
||||||
|
assert.match(h.state.error, /current preview denied/);
|
||||||
|
assert.equal(h.state.detailLoading, false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("initial/loading catalogue does not issue duplicate detail reads; scoped rendering cannot flash old rows", () => {
|
||||||
|
const h = harness();
|
||||||
|
h.state.loading = true;
|
||||||
|
const detail = h.startDetail();
|
||||||
|
assert.equal(detail.cleanup, undefined);
|
||||||
|
assert.equal(h.state.preview, null);
|
||||||
|
assert.match(page, /useMemo\(\(\) => \(\{\}\), \[selectedDatasourceRef, dialogScope, detailRevision\]\)/);
|
||||||
|
assert.match(page, /preview=\{!loading && detailResponseScope === detailScope \? preview : null\}/);
|
||||||
|
assert.match(page, /materializations=\{!loading && detailResponseScope === detailScope \? materializations : \[\]\}/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("older catalogue reloads and revoked authority cannot invalidate or replace the current catalogue", async () => {
|
||||||
|
const h = harness();
|
||||||
|
const older = deferred();
|
||||||
|
h.bindings.listDatasources = () => older.promise;
|
||||||
|
const pending = h.reload();
|
||||||
|
h.bindings.listDatasources = async () => [{ ref: "source-1", revision: 2 }];
|
||||||
|
await h.reload();
|
||||||
|
older.resolve([{ ref: "source-1", revision: 1 }]);
|
||||||
|
await pending;
|
||||||
|
assert.equal(h.state.datasources[0].revision, 2);
|
||||||
|
assert.equal(h.state.detailRevision, 1);
|
||||||
|
const revoked = deferred();
|
||||||
|
h.bindings.listDatasources = () => revoked.promise;
|
||||||
|
const revokedPending = h.reload();
|
||||||
|
h.bindings.authority.current = "authority-B";
|
||||||
|
revoked.resolve([{ ref: "source-1", revision: 3 }]);
|
||||||
|
await revokedPending;
|
||||||
|
assert.equal(h.state.datasources[0].revision, 2);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("an old reload invoked only after B is active cannot issue requests or invalidate B state", async () => {
|
||||||
|
const h = harness();
|
||||||
|
const oldReload = h.reload;
|
||||||
|
h.bindings.authority.current = "authority-B";
|
||||||
|
h.bindings.authorityEpoch.current.revision = 1;
|
||||||
|
h.state.catalogueAuthority = 1;
|
||||||
|
h.state.datasources = [{ ref: "B-source" }];
|
||||||
|
h.state.detailRevision = 8;
|
||||||
|
let calls = 0;
|
||||||
|
h.bindings.listDatasources = async () => { calls += 1; return [{ ref: "A-source" }]; };
|
||||||
|
const requestId = h.bindings.reloadRequestId.current;
|
||||||
|
await oldReload();
|
||||||
|
assert.equal(calls, 0);
|
||||||
|
assert.equal(h.bindings.reloadRequestId.current, requestId);
|
||||||
|
assert.equal(h.state.datasources[0].ref, "B-source");
|
||||||
|
assert.equal(h.state.detailRevision, 8);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("the first authority-transition render cannot request details from a stale catalogue", () => {
|
||||||
|
const h = harness();
|
||||||
|
h.bindings.authority.current = "authority-B";
|
||||||
|
h.bindings.authorityEpoch.current.revision = 1;
|
||||||
|
h.state.loading = false; // The new reload's effect has not committed loading=true yet.
|
||||||
|
const detail = h.startDetail();
|
||||||
|
assert.equal(detail.cleanup, undefined);
|
||||||
|
assert.equal(h.state.detailLoading, false);
|
||||||
|
assert.match(page, /const datasources = catalogueReady \? loadedDatasources : \[\]/);
|
||||||
|
assert.match(page, /const stages = catalogueReady \? loadedStages : \[\]/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("real-authority setup closes confirmations and clears freeze input; unmount blocks old callbacks", () => {
|
||||||
|
const h = harness();
|
||||||
|
const body = extract(" useEffect(() => {\n authority.current = authorityKey;", " }, [reload]);");
|
||||||
|
const flags = ["addOpen", "freezeOpen", "retireOpen", "promoteOpen", "governanceOpen", "decisionOpen", "retentionOpen"];
|
||||||
|
for (const flag of flags) {
|
||||||
|
h.state[flag] = true;
|
||||||
|
// Hide the old intent already in the first transition render, before the
|
||||||
|
// passive effect clears it; it cannot be clicked against a new selection.
|
||||||
|
assert.ok(page.includes(`open={${flag} && catalogueReady}`), flag);
|
||||||
|
}
|
||||||
|
h.state.freezeLabel = "Evidence from prior authority";
|
||||||
|
let reloads = 0;
|
||||||
|
const cleanup = evaluate(`authority.current = authorityKey;${body}`, {
|
||||||
|
...h.bindings, reload: () => { reloads += 1; }
|
||||||
|
});
|
||||||
|
assert.equal(reloads, 1);
|
||||||
|
for (const flag of flags) assert.equal(h.state[flag], false, flag);
|
||||||
|
assert.equal(h.state.freezeLabel, "");
|
||||||
|
assert.equal(h.bindings.isCurrentAuthority(), true);
|
||||||
|
cleanup();
|
||||||
|
assert.equal(h.bindings.isCurrentAuthority(), false);
|
||||||
|
// The reload callback/effect depend on the stable numeric authority epoch,
|
||||||
|
// not cosmetic settings/auth object identities.
|
||||||
|
assert.match(page, /\}, \[dialogScope\]\);\s*useEffect/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("stale mutation success/failure/finally cannot reload, select or overwrite B messages", async () => {
|
||||||
|
for (const operation of ["promoteStage", "refreshSelected", "freezeSelected", "retireSelected"]) {
|
||||||
|
for (const outcome of ["resolve", "reject"]) {
|
||||||
|
const h = harness();
|
||||||
|
const pending = deferred();
|
||||||
|
let reloads = 0;
|
||||||
|
const signature = operation === "freezeSelected" ? "async (): Promise<boolean>" : "async ()";
|
||||||
|
const body = extract(` const ${operation} = ${signature} => {`, "\n };");
|
||||||
|
const result = evaluate(`return (async () => {${body}})();`, {
|
||||||
|
...h.bindings,
|
||||||
|
selectedDatasource: { ref: "A-source", governance: { approval_policy: { required: false } } },
|
||||||
|
selectedStage: { ref: "A-stage", state: "ready" }, freezeLabel: "Freeze A",
|
||||||
|
reload: async () => { reloads += 1; },
|
||||||
|
promoteDatasourceStage: () => pending.promise, refreshDatasource: () => pending.promise,
|
||||||
|
freezeDatasource: () => pending.promise, retireDatasource: () => pending.promise
|
||||||
|
});
|
||||||
|
h.bindings.authority.current = "authority-B";
|
||||||
|
Object.assign(h.state, { success: "B success", error: "B error", view: "origins", working: true });
|
||||||
|
if (outcome === "reject") pending.reject(new Error("A error"));
|
||||||
|
else pending.resolve({ datasource: { ref: "A-source", name: "A" }, materialization: { revision: 1 }, revision: 1 });
|
||||||
|
await result;
|
||||||
|
assert.equal(reloads, 0, operation);
|
||||||
|
assert.equal(h.state.success, "B success", operation);
|
||||||
|
assert.equal(h.state.error, "B error", operation);
|
||||||
|
assert.equal(h.state.view, "origins", operation);
|
||||||
|
assert.equal(h.state.working, true, operation);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("CSV stage dialog sends its explicit mode and exact text; JSON rows remain independent", async () => {
|
||||||
|
const dialog = page.slice(page.indexOf("function AddDatasourceDialog("));
|
||||||
|
const start = dialog.indexOf(" const create = async (): Promise<boolean> => {");
|
||||||
|
const end = dialog.indexOf("\n };", start);
|
||||||
|
assert.ok(start >= 0 && end > start);
|
||||||
|
const body = dialog.slice(start + " const create = async (): Promise<boolean> => {".length, end);
|
||||||
|
const csvText = 'code;value\r\n001;" text "\r\n';
|
||||||
|
for (const format of ["csv", "json"]) {
|
||||||
|
for (const csvValueMode of ["text", "legacy_typed"]) {
|
||||||
|
let payload;
|
||||||
|
const result = await evaluate(`return (async () => {${body}})();`, {
|
||||||
|
kind: "upload", settings: {}, format, csvValueMode, csvText, delimiter: ";",
|
||||||
|
mode: "static", name: "Fixture", sourceName: "fixture", description: "", targetRef: "",
|
||||||
|
rowsText: '[{"code":"001","value":" text "}]', parseRows: JSON.parse,
|
||||||
|
createDatasourceStage: async (_settings, value) => { payload = value; return {}; },
|
||||||
|
onCreated: async () => {}, setBusy: () => {}, setError: () => {}, apiErrorMessage: String
|
||||||
|
});
|
||||||
|
assert.equal(result, true);
|
||||||
|
if (format === "csv") {
|
||||||
|
assert.equal(payload.csv_value_mode, csvValueMode);
|
||||||
|
assert.equal(payload.csv_text, csvText);
|
||||||
|
} else {
|
||||||
|
assert.equal("csv_value_mode" in payload, false);
|
||||||
|
assert.deepEqual(payload.rows, [{ code: "001", value: " text " }]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert.match(dialog, /\[csvValueMode, setCsvValueMode\] = useState<"text" \| "legacy_typed">\("text"\)/);
|
||||||
|
assert.match(dialog, /setCsvValueMode\("text"\)/);
|
||||||
|
assert.match(dialog, /csvValueMode: "text"/);
|
||||||
|
});
|
||||||
@@ -28,11 +28,15 @@ export type DatasourceGovernance = {
|
|||||||
classification: string;
|
classification: string;
|
||||||
privacy_profile_ref?: string | null;
|
privacy_profile_ref?: string | null;
|
||||||
retention_policy_ref?: string | null;
|
retention_policy_ref?: string | null;
|
||||||
|
access_policy_ref?: string | null;
|
||||||
|
visibility_policy: Record<string, unknown>;
|
||||||
hold_refs: string[];
|
hold_refs: string[];
|
||||||
publication_state: string;
|
publication_state: string;
|
||||||
transfer_agreement_ref?: string | null;
|
transfer_agreement_ref?: string | null;
|
||||||
freshness_policy: Record<string, unknown>;
|
freshness_policy: Record<string, unknown>;
|
||||||
quality_policy: Record<string, unknown>;
|
quality_policy: Record<string, unknown>;
|
||||||
|
approval_policy: Record<string, unknown>;
|
||||||
|
retention_policy: Record<string, unknown>;
|
||||||
known_limits: string[];
|
known_limits: string[];
|
||||||
correction_procedure_ref?: string | null;
|
correction_procedure_ref?: string | null;
|
||||||
affected_refs: string[];
|
affected_refs: string[];
|
||||||
@@ -43,6 +47,7 @@ export type DatasourceField = {
|
|||||||
name: string;
|
name: string;
|
||||||
data_type: string;
|
data_type: string;
|
||||||
nullable: boolean;
|
nullable: boolean;
|
||||||
|
classification: string;
|
||||||
};
|
};
|
||||||
|
|
||||||
export type Datasource = {
|
export type Datasource = {
|
||||||
@@ -82,6 +87,8 @@ export type DatasourceMaterialization = {
|
|||||||
frozen_label?: string | null;
|
frozen_label?: string | null;
|
||||||
source_timestamp?: string | null;
|
source_timestamp?: string | null;
|
||||||
created_at?: string | null;
|
created_at?: string | null;
|
||||||
|
disposed_at?: string | null;
|
||||||
|
disposition: Record<string, unknown>;
|
||||||
provenance: Record<string, unknown>;
|
provenance: Record<string, unknown>;
|
||||||
metadata: Record<string, unknown>;
|
metadata: Record<string, unknown>;
|
||||||
governance: DatasourceGovernance;
|
governance: DatasourceGovernance;
|
||||||
@@ -139,6 +146,21 @@ export type DatasourceStage = {
|
|||||||
row_count?: number | null;
|
row_count?: number | null;
|
||||||
byte_count?: number | null;
|
byte_count?: number | null;
|
||||||
validation: DatasourceStageValidation;
|
validation: DatasourceStageValidation;
|
||||||
|
approval: {
|
||||||
|
state?: "not_required" | "pending" | "approved" | "rejected" | "expired";
|
||||||
|
policy_hash?: string;
|
||||||
|
subject_digest?: string;
|
||||||
|
required_approvals?: number;
|
||||||
|
approval_count?: number;
|
||||||
|
expires_at?: string | null;
|
||||||
|
approvals?: Array<{
|
||||||
|
actor_ref?: string;
|
||||||
|
decision?: "approve" | "reject";
|
||||||
|
reason?: string;
|
||||||
|
decided_at?: string;
|
||||||
|
}>;
|
||||||
|
policy?: Record<string, unknown>;
|
||||||
|
};
|
||||||
created_at?: string | null;
|
created_at?: string | null;
|
||||||
promoted_at?: string | null;
|
promoted_at?: string | null;
|
||||||
promoted_materialization_ref?: string | null;
|
promoted_materialization_ref?: string | null;
|
||||||
@@ -200,6 +222,38 @@ export type DatasourcePreview = {
|
|||||||
}>;
|
}>;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
export type DatasourceLifecycleEvidence = {
|
||||||
|
ref: string;
|
||||||
|
subject_ref: string;
|
||||||
|
event_type: string;
|
||||||
|
occurred_at: string;
|
||||||
|
actor_ref?: string | null;
|
||||||
|
policy_version?: string | null;
|
||||||
|
policy_hash?: string | null;
|
||||||
|
subject_digest: string;
|
||||||
|
previous_event_hash?: string | null;
|
||||||
|
event_hash: string;
|
||||||
|
details: Record<string, unknown>;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type DatasourceRetentionCandidate = {
|
||||||
|
ref: string;
|
||||||
|
kind: "stage" | "materialization";
|
||||||
|
datasource_ref?: string | null;
|
||||||
|
disposition: "delete_stage" | "purge_materialization_payload";
|
||||||
|
eligible_at: string;
|
||||||
|
eligible: boolean;
|
||||||
|
blockers: string[];
|
||||||
|
policy_version: string;
|
||||||
|
policy_hash: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type DatasourceRetentionPlan = {
|
||||||
|
as_of: string;
|
||||||
|
plan_hash: string;
|
||||||
|
candidates: DatasourceRetentionCandidate[];
|
||||||
|
};
|
||||||
|
|
||||||
export async function listDatasources(
|
export async function listDatasources(
|
||||||
settings: ApiSettings,
|
settings: ApiSettings,
|
||||||
query = "",
|
query = "",
|
||||||
@@ -270,7 +324,7 @@ export function createDatasourceStage(
|
|||||||
governance?: DatasourceGovernance | null;
|
governance?: DatasourceGovernance | null;
|
||||||
} & (
|
} & (
|
||||||
{ format: "json"; rows: Record<string, unknown>[] }
|
{ format: "json"; rows: Record<string, unknown>[] }
|
||||||
| { format: "csv"; csv_text: string; delimiter: string }
|
| { format: "csv"; csv_text: string; delimiter: string; csv_value_mode?: "text" | "legacy_typed" }
|
||||||
)
|
)
|
||||||
): Promise<DatasourceStage> {
|
): Promise<DatasourceStage> {
|
||||||
return apiFetch(settings, "/api/v1/datasources/stages", {
|
return apiFetch(settings, "/api/v1/datasources/stages", {
|
||||||
@@ -290,6 +344,22 @@ export function promoteDatasourceStage(
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function decideDatasourceStage(
|
||||||
|
settings: ApiSettings,
|
||||||
|
stageRef: string,
|
||||||
|
payload: {
|
||||||
|
decision: "approve" | "reject";
|
||||||
|
reason: string;
|
||||||
|
expected_policy_hash: string;
|
||||||
|
expected_subject_digest: string;
|
||||||
|
}
|
||||||
|
): Promise<DatasourceStage> {
|
||||||
|
return apiFetch(settings, `/api/v1/datasources/stages/${refId(stageRef)}/decision`, {
|
||||||
|
method: "POST",
|
||||||
|
body: JSON.stringify(payload)
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
export function registerDatasourceOrigin(
|
export function registerDatasourceOrigin(
|
||||||
settings: ApiSettings,
|
settings: ApiSettings,
|
||||||
payload: {
|
payload: {
|
||||||
@@ -316,6 +386,50 @@ export function refreshDatasource(
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function prepareDatasourceRefresh(
|
||||||
|
settings: ApiSettings,
|
||||||
|
datasourceRef: string
|
||||||
|
): Promise<DatasourceStage> {
|
||||||
|
return apiFetch(settings, `/api/v1/datasources/${refId(datasourceRef)}/refresh/stage`, {
|
||||||
|
method: "POST"
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function listDatasourceLifecycleEvidence(
|
||||||
|
settings: ApiSettings,
|
||||||
|
subjectRef?: string
|
||||||
|
): Promise<DatasourceLifecycleEvidence[]> {
|
||||||
|
const params = new URLSearchParams();
|
||||||
|
if (subjectRef) params.set("subject_ref", subjectRef);
|
||||||
|
const suffix = params.size ? `?${params.toString()}` : "";
|
||||||
|
const response = await apiFetch<{ evidence: DatasourceLifecycleEvidence[] }>(
|
||||||
|
settings,
|
||||||
|
`/api/v1/datasources/lifecycle-evidence${suffix}`
|
||||||
|
);
|
||||||
|
return response.evidence;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function previewDatasourceRetention(
|
||||||
|
settings: ApiSettings
|
||||||
|
): Promise<DatasourceRetentionPlan> {
|
||||||
|
return apiFetch(settings, "/api/v1/datasources/retention/plan");
|
||||||
|
}
|
||||||
|
|
||||||
|
export function applyDatasourceRetention(
|
||||||
|
settings: ApiSettings,
|
||||||
|
plan: DatasourceRetentionPlan,
|
||||||
|
targetRefs: string[]
|
||||||
|
): Promise<{ plan_hash: string; disposed_refs: string[]; evidence_hashes: string[] }> {
|
||||||
|
return apiFetch(settings, "/api/v1/datasources/retention/apply", {
|
||||||
|
method: "POST",
|
||||||
|
body: JSON.stringify({
|
||||||
|
as_of: plan.as_of,
|
||||||
|
plan_hash: plan.plan_hash,
|
||||||
|
target_refs: targetRefs
|
||||||
|
})
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
export function freezeDatasource(
|
export function freezeDatasource(
|
||||||
settings: ApiSettings,
|
settings: ApiSettings,
|
||||||
datasourceRef: string,
|
datasourceRef: string,
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ import {
|
|||||||
useCallback,
|
useCallback,
|
||||||
useEffect,
|
useEffect,
|
||||||
useMemo,
|
useMemo,
|
||||||
|
useRef,
|
||||||
useState
|
useState
|
||||||
} from "react";
|
} from "react";
|
||||||
import {
|
import {
|
||||||
@@ -15,6 +16,7 @@ import {
|
|||||||
Plus,
|
Plus,
|
||||||
Snowflake,
|
Snowflake,
|
||||||
ShieldCheck,
|
ShieldCheck,
|
||||||
|
ShieldQuestion,
|
||||||
Trash2,
|
Trash2,
|
||||||
Upload
|
Upload
|
||||||
} from "lucide-react";
|
} from "lucide-react";
|
||||||
@@ -42,6 +44,7 @@ import { FormGrid, DialogSection, ActionToolbar,
|
|||||||
WorkspaceFrame,
|
WorkspaceFrame,
|
||||||
WorkspaceLayout,
|
WorkspaceLayout,
|
||||||
hasScope,
|
hasScope,
|
||||||
|
authAuthorityKey,
|
||||||
isApiError,
|
isApiError,
|
||||||
useUnsavedChanges,
|
useUnsavedChanges,
|
||||||
useUnsavedDraftGuard,
|
useUnsavedDraftGuard,
|
||||||
@@ -49,14 +52,18 @@ import { FormGrid, DialogSection, ActionToolbar,
|
|||||||
type AuthInfo
|
type AuthInfo
|
||||||
} from "@govoplan/core-webui";
|
} from "@govoplan/core-webui";
|
||||||
import {
|
import {
|
||||||
|
applyDatasourceRetention,
|
||||||
createDatasourceStage,
|
createDatasourceStage,
|
||||||
|
decideDatasourceStage,
|
||||||
freezeDatasource,
|
freezeDatasource,
|
||||||
listDatasourceMaterializations,
|
listDatasourceMaterializations,
|
||||||
listDatasourceOrigins,
|
listDatasourceOrigins,
|
||||||
listDatasourceStages,
|
listDatasourceStages,
|
||||||
listDatasources,
|
listDatasources,
|
||||||
previewDatasource,
|
previewDatasource,
|
||||||
|
previewDatasourceRetention,
|
||||||
promoteDatasourceStage,
|
promoteDatasourceStage,
|
||||||
|
prepareDatasourceRefresh,
|
||||||
refreshDatasource,
|
refreshDatasource,
|
||||||
registerDatasourceOrigin,
|
registerDatasourceOrigin,
|
||||||
retireDatasource,
|
retireDatasource,
|
||||||
@@ -66,6 +73,7 @@ import {
|
|||||||
type DatasourceMaterialization,
|
type DatasourceMaterialization,
|
||||||
type DatasourceOrigin,
|
type DatasourceOrigin,
|
||||||
type DatasourcePreview,
|
type DatasourcePreview,
|
||||||
|
type DatasourceRetentionPlan,
|
||||||
type DatasourceSchemaChange,
|
type DatasourceSchemaChange,
|
||||||
type DatasourceStage,
|
type DatasourceStage,
|
||||||
type DatasourceValidationDiagnostic
|
type DatasourceValidationDiagnostic
|
||||||
@@ -73,6 +81,7 @@ import {
|
|||||||
import {
|
import {
|
||||||
DATASOURCE_FIELDS_DOCUMENTATION,
|
DATASOURCE_FIELDS_DOCUMENTATION,
|
||||||
DATASOURCE_GOVERNANCE_DOCUMENTATION,
|
DATASOURCE_GOVERNANCE_DOCUMENTATION,
|
||||||
|
DATASOURCE_VISIBILITY_DOCUMENTATION,
|
||||||
DATASOURCES_DOCUMENTATION,
|
DATASOURCES_DOCUMENTATION,
|
||||||
DATASOURCES_I18N
|
DATASOURCES_I18N
|
||||||
} from "./interfacePatterns";
|
} from "./interfacePatterns";
|
||||||
@@ -89,10 +98,10 @@ export default function DatasourcesPage({
|
|||||||
auth: AuthInfo;
|
auth: AuthInfo;
|
||||||
}) {
|
}) {
|
||||||
const [view, setView] = useState<CatalogueView>("catalogue");
|
const [view, setView] = useState<CatalogueView>("catalogue");
|
||||||
const [datasources, setDatasources] = useState<Datasource[]>([]);
|
const [loadedDatasources, setDatasources] = useState<Datasource[]>([]);
|
||||||
const [stages, setStages] = useState<DatasourceStage[]>([]);
|
const [loadedStages, setStages] = useState<DatasourceStage[]>([]);
|
||||||
const [origins, setOrigins] = useState<DatasourceOrigin[]>([]);
|
const [loadedOrigins, setOrigins] = useState<DatasourceOrigin[]>([]);
|
||||||
const [originsAvailable, setOriginsAvailable] = useState(false);
|
const [loadedOriginsAvailable, setOriginsAvailable] = useState(false);
|
||||||
const [selectedDatasourceRef, setSelectedDatasourceRef] = useState(
|
const [selectedDatasourceRef, setSelectedDatasourceRef] = useState(
|
||||||
initialDatasourceRef
|
initialDatasourceRef
|
||||||
);
|
);
|
||||||
@@ -103,6 +112,28 @@ export default function DatasourcesPage({
|
|||||||
const [search, setSearch] = useState("");
|
const [search, setSearch] = useState("");
|
||||||
const [loading, setLoading] = useState(true);
|
const [loading, setLoading] = useState(true);
|
||||||
const [detailLoading, setDetailLoading] = useState(false);
|
const [detailLoading, setDetailLoading] = useState(false);
|
||||||
|
const [detailRevision, setDetailRevision] = useState(0);
|
||||||
|
const authorityKey = authAuthorityKey(auth, settings);
|
||||||
|
const authority = useRef(authorityKey);
|
||||||
|
authority.current = authorityKey;
|
||||||
|
const authorityEpoch = useRef({ key: authorityKey, revision: 0 });
|
||||||
|
if (authorityEpoch.current.key !== authorityKey) {
|
||||||
|
authorityEpoch.current = { key: authorityKey, revision: authorityEpoch.current.revision + 1 };
|
||||||
|
}
|
||||||
|
const dialogScope = authorityEpoch.current.revision;
|
||||||
|
const isCurrentAuthority = () => authority.current === authorityKey
|
||||||
|
&& authorityEpoch.current.revision === dialogScope;
|
||||||
|
const [catalogueAuthority, setCatalogueAuthority] = useState<number | null>(null);
|
||||||
|
const catalogueReady = catalogueAuthority === dialogScope;
|
||||||
|
const datasources = catalogueReady ? loadedDatasources : [];
|
||||||
|
const stages = catalogueReady ? loadedStages : [];
|
||||||
|
const origins = catalogueReady ? loadedOrigins : [];
|
||||||
|
const originsAvailable = catalogueReady && loadedOriginsAvailable;
|
||||||
|
const detailScope = useMemo(() => ({}), [selectedDatasourceRef, dialogScope, detailRevision]);
|
||||||
|
const currentDetailScope = useRef(detailScope);
|
||||||
|
currentDetailScope.current = detailScope;
|
||||||
|
const [detailResponseScope, setDetailResponseScope] = useState<object | null>(null);
|
||||||
|
const reloadRequestId = useRef(0);
|
||||||
const [working, setWorking] = useState(false);
|
const [working, setWorking] = useState(false);
|
||||||
const [error, setError] = useState("");
|
const [error, setError] = useState("");
|
||||||
const [success, setSuccess] = useState("");
|
const [success, setSuccess] = useState("");
|
||||||
@@ -112,14 +143,25 @@ export default function DatasourcesPage({
|
|||||||
const [retireOpen, setRetireOpen] = useState(false);
|
const [retireOpen, setRetireOpen] = useState(false);
|
||||||
const [promoteOpen, setPromoteOpen] = useState(false);
|
const [promoteOpen, setPromoteOpen] = useState(false);
|
||||||
const [governanceOpen, setGovernanceOpen] = useState(false);
|
const [governanceOpen, setGovernanceOpen] = useState(false);
|
||||||
|
const [decisionOpen, setDecisionOpen] = useState(false);
|
||||||
|
const [retentionOpen, setRetentionOpen] = useState(false);
|
||||||
const { requestDiscard } = useUnsavedChanges();
|
const { requestDiscard } = useUnsavedChanges();
|
||||||
|
|
||||||
const canManage = hasScope(auth, "datasources:source:write")
|
const canManage = hasScope(auth, "datasources:source:write")
|
||||||
|| hasScope(auth, "datasources:source:admin");
|
|| hasScope(auth, "datasources:source:admin");
|
||||||
const canStage = hasScope(auth, "datasources:stage:write")
|
const canStage = hasScope(auth, "datasources:stage:write")
|
||||||
|| hasScope(auth, "datasources:source:admin");
|
|| hasScope(auth, "datasources:source:admin");
|
||||||
|
const canApprove = hasScope(auth, "datasources:stage:approve")
|
||||||
|
|| hasScope(auth, "datasources:source:admin");
|
||||||
|
const canAdmin = hasScope(auth, "datasources:source:admin");
|
||||||
|
|
||||||
const reload = useCallback(async (preferredDatasourceRef?: string) => {
|
const reload = useCallback(async (preferredDatasourceRef?: string) => {
|
||||||
|
// An old mutation callback can invoke this closure AFTER authority changed.
|
||||||
|
// Compare the closure's scope before any request, state write or generation bump.
|
||||||
|
if (!isCurrentAuthority()) return;
|
||||||
|
const requestId = ++reloadRequestId.current;
|
||||||
|
const isCurrent = () => requestId === reloadRequestId.current
|
||||||
|
&& isCurrentAuthority();
|
||||||
setLoading(true);
|
setLoading(true);
|
||||||
setError("");
|
setError("");
|
||||||
try {
|
try {
|
||||||
@@ -128,10 +170,15 @@ export default function DatasourcesPage({
|
|||||||
listDatasourceStages(settings),
|
listDatasourceStages(settings),
|
||||||
listDatasourceOrigins(settings)
|
listDatasourceOrigins(settings)
|
||||||
]);
|
]);
|
||||||
|
if (!isCurrent()) return;
|
||||||
|
setCatalogueAuthority(dialogScope);
|
||||||
setDatasources(nextDatasources);
|
setDatasources(nextDatasources);
|
||||||
setStages(nextStages);
|
setStages(nextStages);
|
||||||
setOrigins(originCatalogue.origins);
|
setOrigins(originCatalogue.origins);
|
||||||
setOriginsAvailable(originCatalogue.available);
|
setOriginsAvailable(originCatalogue.available);
|
||||||
|
// Reload also invalidates history/preview when the selected reference did
|
||||||
|
// not change (refresh, freeze, promotion, governance and manual reload).
|
||||||
|
setDetailRevision((current) => current + 1);
|
||||||
setSelectedDatasourceRef((current) => {
|
setSelectedDatasourceRef((current) => {
|
||||||
const preferred = preferredDatasourceRef || current;
|
const preferred = preferredDatasourceRef || current;
|
||||||
return nextDatasources.some((item) => item.ref === preferred)
|
return nextDatasources.some((item) => item.ref === preferred)
|
||||||
@@ -145,20 +192,38 @@ export default function DatasourcesPage({
|
|||||||
? current
|
? current
|
||||||
: originCatalogue.origins[0]?.ref ?? "");
|
: originCatalogue.origins[0]?.ref ?? "");
|
||||||
} catch (loadError) {
|
} catch (loadError) {
|
||||||
setError(apiErrorMessage(loadError));
|
if (isCurrent()) setError(apiErrorMessage(loadError));
|
||||||
} finally {
|
} finally {
|
||||||
setLoading(false);
|
if (isCurrent()) setLoading(false);
|
||||||
}
|
}
|
||||||
}, [settings]);
|
}, [dialogScope]);
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
|
authority.current = authorityKey;
|
||||||
|
setWorking(false);
|
||||||
|
setSuccess("");
|
||||||
|
// Confirmations and their input must not follow a selection into a new
|
||||||
|
// authority context. Equivalent session refreshes keep this effect stable.
|
||||||
|
setAddOpen(false);
|
||||||
|
setFreezeOpen(false);
|
||||||
|
setFreezeLabel("");
|
||||||
|
setRetireOpen(false);
|
||||||
|
setPromoteOpen(false);
|
||||||
|
setGovernanceOpen(false);
|
||||||
|
setDecisionOpen(false);
|
||||||
|
setRetentionOpen(false);
|
||||||
void reload();
|
void reload();
|
||||||
|
return () => {
|
||||||
|
reloadRequestId.current += 1;
|
||||||
|
if (authority.current === authorityKey) authority.current = "";
|
||||||
|
};
|
||||||
}, [reload]);
|
}, [reload]);
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
if (!selectedDatasourceRef) {
|
if (!selectedDatasourceRef || loading || !catalogueReady) {
|
||||||
setPreview(null);
|
setPreview(null);
|
||||||
setMaterializations([]);
|
setMaterializations([]);
|
||||||
|
setDetailLoading(false);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
let cancelled = false;
|
let cancelled = false;
|
||||||
@@ -167,7 +232,8 @@ export default function DatasourcesPage({
|
|||||||
previewDatasource(settings, selectedDatasourceRef),
|
previewDatasource(settings, selectedDatasourceRef),
|
||||||
listDatasourceMaterializations(settings, selectedDatasourceRef)
|
listDatasourceMaterializations(settings, selectedDatasourceRef)
|
||||||
]).then(([previewResult, materializationResult]) => {
|
]).then(([previewResult, materializationResult]) => {
|
||||||
if (cancelled) return;
|
if (cancelled || currentDetailScope.current !== detailScope) return;
|
||||||
|
setDetailResponseScope(detailScope);
|
||||||
if (previewResult.status === "fulfilled") {
|
if (previewResult.status === "fulfilled") {
|
||||||
setPreview(previewResult.value);
|
setPreview(previewResult.value);
|
||||||
} else {
|
} else {
|
||||||
@@ -184,7 +250,7 @@ export default function DatasourcesPage({
|
|||||||
return () => {
|
return () => {
|
||||||
cancelled = true;
|
cancelled = true;
|
||||||
};
|
};
|
||||||
}, [selectedDatasourceRef, settings]);
|
}, [selectedDatasourceRef, detailScope, loading, catalogueReady]);
|
||||||
|
|
||||||
const selectedDatasource = datasources.find((item) => item.ref === selectedDatasourceRef) ?? null;
|
const selectedDatasource = datasources.find((item) => item.ref === selectedDatasourceRef) ?? null;
|
||||||
const selectedStage = stages.find((item) => item.ref === selectedStageRef) ?? null;
|
const selectedStage = stages.find((item) => item.ref === selectedStageRef) ?? null;
|
||||||
@@ -207,38 +273,50 @@ export default function DatasourcesPage({
|
|||||||
);
|
);
|
||||||
|
|
||||||
const promoteStage = async () => {
|
const promoteStage = async () => {
|
||||||
if (!selectedStage || selectedStage.state !== "ready") return;
|
if (!isCurrentAuthority() || !selectedStage || selectedStage.state !== "ready") return;
|
||||||
setWorking(true);
|
setWorking(true);
|
||||||
setError("");
|
setError("");
|
||||||
try {
|
try {
|
||||||
const result = await promoteDatasourceStage(settings, selectedStage.ref);
|
const result = await promoteDatasourceStage(settings, selectedStage.ref);
|
||||||
|
if (!isCurrentAuthority()) return;
|
||||||
setSuccess(`Promoted ${result.datasource.name} as revision ${result.materialization.revision}.`);
|
setSuccess(`Promoted ${result.datasource.name} as revision ${result.materialization.revision}.`);
|
||||||
setView("catalogue");
|
setView("catalogue");
|
||||||
await reload(result.datasource.ref);
|
await reload(result.datasource.ref);
|
||||||
} catch (operationError) {
|
} catch (operationError) {
|
||||||
setError(apiErrorMessage(operationError));
|
if (isCurrentAuthority()) setError(apiErrorMessage(operationError));
|
||||||
} finally {
|
} finally {
|
||||||
setWorking(false);
|
if (isCurrentAuthority()) setWorking(false);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
const refreshSelected = async () => {
|
const refreshSelected = async () => {
|
||||||
if (!selectedDatasource) return;
|
if (!isCurrentAuthority() || !selectedDatasource) return;
|
||||||
setWorking(true);
|
setWorking(true);
|
||||||
setError("");
|
setError("");
|
||||||
try {
|
try {
|
||||||
|
if (selectedDatasource.governance.approval_policy.required === true) {
|
||||||
|
const stage = await prepareDatasourceRefresh(settings, selectedDatasource.ref);
|
||||||
|
if (!isCurrentAuthority()) return;
|
||||||
|
await reload(selectedDatasource.ref);
|
||||||
|
if (!isCurrentAuthority()) return;
|
||||||
|
setSelectedStageRef(stage.ref);
|
||||||
|
setView("staging");
|
||||||
|
setSuccess(`Prepared refresh stage ${stage.name}. It must be approved before promotion.`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
const result = await refreshDatasource(settings, selectedDatasource.ref);
|
const result = await refreshDatasource(settings, selectedDatasource.ref);
|
||||||
|
if (!isCurrentAuthority()) return;
|
||||||
setSuccess(`Refreshed ${result.datasource.name} as revision ${result.materialization.revision}.`);
|
setSuccess(`Refreshed ${result.datasource.name} as revision ${result.materialization.revision}.`);
|
||||||
await reload(result.datasource.ref);
|
await reload(result.datasource.ref);
|
||||||
} catch (operationError) {
|
} catch (operationError) {
|
||||||
setError(apiErrorMessage(operationError));
|
if (isCurrentAuthority()) setError(apiErrorMessage(operationError));
|
||||||
} finally {
|
} finally {
|
||||||
setWorking(false);
|
if (isCurrentAuthority()) setWorking(false);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
const freezeSelected = async (): Promise<boolean> => {
|
const freezeSelected = async (): Promise<boolean> => {
|
||||||
if (!selectedDatasource) return false;
|
if (!isCurrentAuthority() || !selectedDatasource) return false;
|
||||||
setWorking(true);
|
setWorking(true);
|
||||||
setError("");
|
setError("");
|
||||||
try {
|
try {
|
||||||
@@ -247,32 +325,34 @@ export default function DatasourcesPage({
|
|||||||
selectedDatasource.ref,
|
selectedDatasource.ref,
|
||||||
freezeLabel
|
freezeLabel
|
||||||
);
|
);
|
||||||
|
if (!isCurrentAuthority()) return false;
|
||||||
setSuccess(`Created frozen revision ${materialization.revision}.`);
|
setSuccess(`Created frozen revision ${materialization.revision}.`);
|
||||||
setFreezeOpen(false);
|
setFreezeOpen(false);
|
||||||
setFreezeLabel("");
|
setFreezeLabel("");
|
||||||
await reload(selectedDatasource.ref);
|
await reload(selectedDatasource.ref);
|
||||||
return true;
|
return isCurrentAuthority();
|
||||||
} catch (operationError) {
|
} catch (operationError) {
|
||||||
setError(apiErrorMessage(operationError));
|
if (isCurrentAuthority()) setError(apiErrorMessage(operationError));
|
||||||
return false;
|
return false;
|
||||||
} finally {
|
} finally {
|
||||||
setWorking(false);
|
if (isCurrentAuthority()) setWorking(false);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
const retireSelected = async () => {
|
const retireSelected = async () => {
|
||||||
if (!selectedDatasource) return;
|
if (!isCurrentAuthority() || !selectedDatasource) return;
|
||||||
setWorking(true);
|
setWorking(true);
|
||||||
setError("");
|
setError("");
|
||||||
try {
|
try {
|
||||||
await retireDatasource(settings, selectedDatasource.ref);
|
await retireDatasource(settings, selectedDatasource.ref);
|
||||||
|
if (!isCurrentAuthority()) return;
|
||||||
setSuccess(`Retired ${selectedDatasource.name}.`);
|
setSuccess(`Retired ${selectedDatasource.name}.`);
|
||||||
setRetireOpen(false);
|
setRetireOpen(false);
|
||||||
await reload();
|
await reload();
|
||||||
} catch (operationError) {
|
} catch (operationError) {
|
||||||
setError(apiErrorMessage(operationError));
|
if (isCurrentAuthority()) setError(apiErrorMessage(operationError));
|
||||||
} finally {
|
} finally {
|
||||||
setWorking(false);
|
if (isCurrentAuthority()) setWorking(false);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -295,6 +375,22 @@ export default function DatasourcesPage({
|
|||||||
|
|
||||||
return (
|
return (
|
||||||
<WorkspaceFrame as="main" height="viewport" surface="plain" className="datasources-page" label="Datasource workspace">
|
<WorkspaceFrame as="main" height="viewport" surface="plain" className="datasources-page" label="Datasource workspace">
|
||||||
|
<WorkspaceActionBar
|
||||||
|
title="Data"
|
||||||
|
titleHelp={<DocumentationHelpLink reference={DATASOURCES_DOCUMENTATION} />}
|
||||||
|
scope="workspace"
|
||||||
|
variant="collection"
|
||||||
|
refreshable
|
||||||
|
reloadAction={{ onReload: () => void reload(selectedDatasourceRef), loading: loading || working }}
|
||||||
|
createAction={<IconButton
|
||||||
|
label="Add datasource or stage"
|
||||||
|
icon={<Plus size={17} />}
|
||||||
|
variant="primary"
|
||||||
|
onClick={() => setAddOpen(true)}
|
||||||
|
disabled={!canStage && !canManage}
|
||||||
|
disabledReason={!canStage && !canManage ? DATASOURCES_I18N.manageReason : undefined}
|
||||||
|
/>}
|
||||||
|
/>
|
||||||
<WorkspaceLayout
|
<WorkspaceLayout
|
||||||
variant="split"
|
variant="split"
|
||||||
primarySize="compact"
|
primarySize="compact"
|
||||||
@@ -305,21 +401,6 @@ export default function DatasourcesPage({
|
|||||||
contentLabel="Datasource workspace"
|
contentLabel="Datasource workspace"
|
||||||
contentClassName="datasources-workspace"
|
contentClassName="datasources-workspace"
|
||||||
primary={<>
|
primary={<>
|
||||||
<WorkspaceActionBar
|
|
||||||
scope="collection-pane"
|
|
||||||
variant="collection"
|
|
||||||
refreshable
|
|
||||||
reloadAction={{ onReload: () => void reload(selectedDatasourceRef), loading: loading || working }}
|
|
||||||
contextActions={<strong>Data</strong>}
|
|
||||||
createAction={<IconButton
|
|
||||||
label="Add datasource or stage"
|
|
||||||
icon={<Plus size={17} />}
|
|
||||||
variant="primary"
|
|
||||||
onClick={() => setAddOpen(true)}
|
|
||||||
disabled={!canStage && !canManage}
|
|
||||||
disabledReason={!canStage && !canManage ? DATASOURCES_I18N.manageReason : undefined}
|
|
||||||
/>}
|
|
||||||
/>
|
|
||||||
<div className="datasources-view-switch">
|
<div className="datasources-view-switch">
|
||||||
<SegmentedControl<CatalogueView>
|
<SegmentedControl<CatalogueView>
|
||||||
ariaLabel="Datasource view"
|
ariaLabel="Datasource view"
|
||||||
@@ -420,11 +501,15 @@ export default function DatasourcesPage({
|
|||||||
</small>
|
</small>
|
||||||
</span>
|
</span>
|
||||||
</span>}
|
</span>}
|
||||||
helpAction={<DocumentationHelpLink reference={DATASOURCES_DOCUMENTATION} />}
|
|
||||||
primaryActions={<>
|
primaryActions={<>
|
||||||
{view === "catalogue" && selectedDatasource?.mode === "cached" ? (
|
{view === "catalogue" && selectedDatasource?.mode === "cached" ? (
|
||||||
<Button onClick={() => void refreshSelected()} disabled={!canManage || working} disabledReason={working ? DATASOURCES_I18N.working : !canManage ? DATASOURCES_I18N.manageReason : undefined}>
|
<Button onClick={() => void refreshSelected()} disabled={!canManage || working} disabledReason={working ? DATASOURCES_I18N.working : !canManage ? DATASOURCES_I18N.manageReason : undefined}>
|
||||||
<Download size={16} /> Refresh
|
<Download size={16} /> {selectedDatasource.governance.approval_policy.required === true ? "Stage refresh" : "Refresh"}
|
||||||
|
</Button>
|
||||||
|
) : null}
|
||||||
|
{view === "catalogue" && canAdmin ? (
|
||||||
|
<Button helpContextId="datasources.retention" helpModuleId="datasources" onClick={() => setRetentionOpen(true)} disabled={working}>
|
||||||
|
<Archive size={16} /> Retention
|
||||||
</Button>
|
</Button>
|
||||||
) : null}
|
) : null}
|
||||||
{view === "catalogue" && selectedDatasource ? (
|
{view === "catalogue" && selectedDatasource ? (
|
||||||
@@ -451,6 +536,16 @@ export default function DatasourcesPage({
|
|||||||
<Upload size={16} /> Promote
|
<Upload size={16} /> Promote
|
||||||
</Button>
|
</Button>
|
||||||
) : null}
|
) : null}
|
||||||
|
{view === "staging" && selectedStage?.state === "awaiting_approval" ? (
|
||||||
|
<Button
|
||||||
|
variant="primary"
|
||||||
|
onClick={() => setDecisionOpen(true)}
|
||||||
|
disabled={!canApprove || working}
|
||||||
|
disabledReason={working ? DATASOURCES_I18N.working : !canApprove ? "Datasource approval permission is required." : undefined}
|
||||||
|
>
|
||||||
|
<ShieldQuestion size={16} /> Decide
|
||||||
|
</Button>
|
||||||
|
) : null}
|
||||||
{view === "origins" && selectedOrigin ? (
|
{view === "origins" && selectedOrigin ? (
|
||||||
<Button
|
<Button
|
||||||
variant="primary"
|
variant="primary"
|
||||||
@@ -504,9 +599,9 @@ export default function DatasourcesPage({
|
|||||||
selectedDatasource ? (
|
selectedDatasource ? (
|
||||||
<DatasourceDetail
|
<DatasourceDetail
|
||||||
datasource={selectedDatasource}
|
datasource={selectedDatasource}
|
||||||
preview={preview}
|
preview={!loading && detailResponseScope === detailScope ? preview : null}
|
||||||
materializations={materializations}
|
materializations={!loading && detailResponseScope === detailScope ? materializations : []}
|
||||||
loading={detailLoading}
|
loading={loading || detailLoading || detailResponseScope !== detailScope}
|
||||||
/>
|
/>
|
||||||
) : <EmptyWorkspace icon={<DatabaseZap size={32} />} label="No datasource selected" />
|
) : <EmptyWorkspace icon={<DatabaseZap size={32} />} label="No datasource selected" />
|
||||||
) : null}
|
) : null}
|
||||||
@@ -527,7 +622,8 @@ export default function DatasourcesPage({
|
|||||||
</WorkspaceLayout>
|
</WorkspaceLayout>
|
||||||
|
|
||||||
<AddDatasourceDialog
|
<AddDatasourceDialog
|
||||||
open={addOpen}
|
key={`add:${dialogScope}`}
|
||||||
|
open={addOpen && catalogueReady}
|
||||||
settings={settings}
|
settings={settings}
|
||||||
initialKind={view === "origins" ? "origin" : "upload"}
|
initialKind={view === "origins" ? "origin" : "upload"}
|
||||||
initialOrigin={view === "origins" ? selectedOrigin : null}
|
initialOrigin={view === "origins" ? selectedOrigin : null}
|
||||||
@@ -538,6 +634,7 @@ export default function DatasourcesPage({
|
|||||||
canManage={canManage}
|
canManage={canManage}
|
||||||
onClose={() => setAddOpen(false)}
|
onClose={() => setAddOpen(false)}
|
||||||
onCreated={async (result) => {
|
onCreated={async (result) => {
|
||||||
|
if (!isCurrentAuthority()) return;
|
||||||
setAddOpen(false);
|
setAddOpen(false);
|
||||||
if ("state" in result) {
|
if ("state" in result) {
|
||||||
setView("staging");
|
setView("staging");
|
||||||
@@ -552,18 +649,47 @@ export default function DatasourcesPage({
|
|||||||
}}
|
}}
|
||||||
/>
|
/>
|
||||||
<GovernanceDialog
|
<GovernanceDialog
|
||||||
open={governanceOpen}
|
key={`governance:${dialogScope}`}
|
||||||
|
open={governanceOpen && catalogueReady}
|
||||||
settings={settings}
|
settings={settings}
|
||||||
datasource={selectedDatasource}
|
datasource={selectedDatasource}
|
||||||
onClose={() => setGovernanceOpen(false)}
|
onClose={() => setGovernanceOpen(false)}
|
||||||
onSaved={async (updated) => {
|
onSaved={async (updated) => {
|
||||||
|
if (!isCurrentAuthority()) return;
|
||||||
setGovernanceOpen(false);
|
setGovernanceOpen(false);
|
||||||
setSuccess(`Updated governance for ${updated.name}.`);
|
setSuccess(`Updated governance for ${updated.name}.`);
|
||||||
await reload(updated.ref);
|
await reload(updated.ref);
|
||||||
}}
|
}}
|
||||||
/>
|
/>
|
||||||
|
<StageDecisionDialog
|
||||||
|
key={`decision:${dialogScope}`}
|
||||||
|
open={decisionOpen && catalogueReady}
|
||||||
|
settings={settings}
|
||||||
|
stage={selectedStage}
|
||||||
|
onClose={() => setDecisionOpen(false)}
|
||||||
|
onDecided={async (updated) => {
|
||||||
|
if (!isCurrentAuthority()) return;
|
||||||
|
setDecisionOpen(false);
|
||||||
|
await reload();
|
||||||
|
if (!isCurrentAuthority()) return;
|
||||||
|
setSelectedStageRef(updated.ref);
|
||||||
|
setSuccess(`Recorded ${updated.approval.state ?? "approval"} decision state for ${updated.name}.`);
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
<RetentionDialog
|
||||||
|
key={`retention:${dialogScope}`}
|
||||||
|
open={retentionOpen && catalogueReady}
|
||||||
|
settings={settings}
|
||||||
|
onClose={() => setRetentionOpen(false)}
|
||||||
|
onApplied={async (count) => {
|
||||||
|
if (!isCurrentAuthority()) return;
|
||||||
|
setRetentionOpen(false);
|
||||||
|
setSuccess(`Applied retention to ${count} eligible target${count === 1 ? "" : "s"}.`);
|
||||||
|
await reload(selectedDatasourceRef);
|
||||||
|
}}
|
||||||
|
/>
|
||||||
<Dialog
|
<Dialog
|
||||||
open={freezeOpen}
|
open={freezeOpen && catalogueReady}
|
||||||
title="Freeze datasource state"
|
title="Freeze datasource state"
|
||||||
onClose={closeFreeze}
|
onClose={closeFreeze}
|
||||||
footer={(
|
footer={(
|
||||||
@@ -587,7 +713,7 @@ export default function DatasourcesPage({
|
|||||||
</FormField>
|
</FormField>
|
||||||
</Dialog>
|
</Dialog>
|
||||||
<ConfirmDialog
|
<ConfirmDialog
|
||||||
open={promoteOpen}
|
open={promoteOpen && catalogueReady}
|
||||||
title="i18n:govoplan-datasources.promote_title"
|
title="i18n:govoplan-datasources.promote_title"
|
||||||
message="i18n:govoplan-datasources.promote_message"
|
message="i18n:govoplan-datasources.promote_message"
|
||||||
confirmLabel="Promote"
|
confirmLabel="Promote"
|
||||||
@@ -599,7 +725,7 @@ export default function DatasourcesPage({
|
|||||||
}}
|
}}
|
||||||
/>
|
/>
|
||||||
<ConfirmDialog
|
<ConfirmDialog
|
||||||
open={retireOpen}
|
open={retireOpen && catalogueReady}
|
||||||
title="Retire datasource"
|
title="Retire datasource"
|
||||||
message={`Retire ${selectedDatasource?.name ?? "this datasource"}? Existing materialization references remain in the database, but the datasource will no longer be available to new definitions.`}
|
message={`Retire ${selectedDatasource?.name ?? "this datasource"}? Existing materialization references remain in the database, but the datasource will no longer be available to new definitions.`}
|
||||||
confirmLabel="Retire"
|
confirmLabel="Retire"
|
||||||
@@ -629,7 +755,7 @@ function DatasourceDetail({
|
|||||||
<MetricCard density="compact" label="Mode" value={datasource.mode} valueTitle={datasource.mode} />
|
<MetricCard density="compact" label="Mode" value={datasource.mode} valueTitle={datasource.mode} />
|
||||||
<MetricCard density="compact" label="Rows" value={formatNumber(datasource.row_count)} />
|
<MetricCard density="compact" label="Rows" value={formatNumber(datasource.row_count)} />
|
||||||
<MetricCard density="compact" label="Fields" value={String(datasource.schema.length)} />
|
<MetricCard density="compact" label="Fields" value={String(datasource.schema.length)} />
|
||||||
<MetricCard density="compact" label="Revisions" value={String(materializations.length)} />
|
<MetricCard density="compact" label="Revisions" value={loading ? "…" : String(materializations.length)} />
|
||||||
<MetricCard density="compact" label="Updated" value={formatDate(datasource.updated_at)} />
|
<MetricCard density="compact" label="Updated" value={formatDate(datasource.updated_at)} />
|
||||||
</MetricGrid>
|
</MetricGrid>
|
||||||
{datasource.description ? (
|
{datasource.description ? (
|
||||||
@@ -650,6 +776,8 @@ function DatasourceDetail({
|
|||||||
<span><small>Steward</small><strong>{datasource.governance.steward_ref || "Not assigned"}</strong></span>
|
<span><small>Steward</small><strong>{datasource.governance.steward_ref || "Not assigned"}</strong></span>
|
||||||
<span><small>Responsible organization</small><strong>{datasource.governance.responsible_organization_ref || "Not assigned"}</strong></span>
|
<span><small>Responsible organization</small><strong>{datasource.governance.responsible_organization_ref || "Not assigned"}</strong></span>
|
||||||
<span><small>Purposes</small><strong>{datasource.governance.purposes.join(", ") || "Not declared"}</strong></span>
|
<span><small>Purposes</small><strong>{datasource.governance.purposes.join(", ") || "Not declared"}</strong></span>
|
||||||
|
<span><small>Approval gate</small><strong>{datasource.governance.approval_policy.required === true ? "Required" : "Not required"}</strong></span>
|
||||||
|
<span><small>Retention</small><strong>{datasource.governance.retention_policy.enabled === true ? "Configured" : "Disabled"}</strong></span>
|
||||||
</div>
|
</div>
|
||||||
{datasource.governance.semantic_definition ? (
|
{datasource.governance.semantic_definition ? (
|
||||||
<p className="datasources-dialog-copy">{datasource.governance.semantic_definition}</p>
|
<p className="datasources-dialog-copy">{datasource.governance.semantic_definition}</p>
|
||||||
@@ -691,7 +819,9 @@ function DatasourceDetail({
|
|||||||
<td>{formatDate(item.created_at)}</td>
|
<td>{formatDate(item.created_at)}</td>
|
||||||
<td>{formatNumber(item.row_count)}</td>
|
<td>{formatNumber(item.row_count)}</td>
|
||||||
<td>
|
<td>
|
||||||
{item.frozen_at ? (
|
{item.disposed_at ? (
|
||||||
|
<StatusBadge status="disposed" label="payload disposed" />
|
||||||
|
) : item.frozen_at ? (
|
||||||
<StatusBadge
|
<StatusBadge
|
||||||
status="frozen"
|
status="frozen"
|
||||||
label={item.frozen_label || "frozen"}
|
label={item.frozen_label || "frozen"}
|
||||||
@@ -703,7 +833,9 @@ function DatasourceDetail({
|
|||||||
</td>
|
</td>
|
||||||
</tr>
|
</tr>
|
||||||
))}
|
))}
|
||||||
{!materializations.length ? (
|
{loading ? (
|
||||||
|
<tr><td colSpan={5}>Loading materializations...</td></tr>
|
||||||
|
) : !materializations.length ? (
|
||||||
<tr><td colSpan={5}>Live source without materializations</td></tr>
|
<tr><td colSpan={5}>Live source without materializations</td></tr>
|
||||||
) : null}
|
) : null}
|
||||||
</tbody>
|
</tbody>
|
||||||
@@ -772,6 +904,28 @@ function StageDetail({ stage }: { stage: DatasourceStage }) {
|
|||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
</ContentSection>
|
</ContentSection>
|
||||||
|
<ContentSection>
|
||||||
|
<ActionToolbar surface="section-header" className="datasources-section-heading">
|
||||||
|
<span><ShieldQuestion size={16} /> Promotion approval</span>
|
||||||
|
<StatusBadge status={stage.approval.state ?? "not_required"} label={readableToken(stage.approval.state ?? "not_required")} />
|
||||||
|
</ActionToolbar>
|
||||||
|
<div className="datasources-key-values">
|
||||||
|
<span><small>Progress</small><strong>{stage.approval.approval_count ?? 0} / {stage.approval.required_approvals ?? 1}</strong></span>
|
||||||
|
<span><small>Policy hash</small><strong>{shortFingerprint(stage.approval.policy_hash ?? "")}</strong></span>
|
||||||
|
<span><small>Subject digest</small><strong>{shortFingerprint(stage.approval.subject_digest ?? "")}</strong></span>
|
||||||
|
<span><small>Expires</small><strong>{formatDate(stage.approval.expires_at)}</strong></span>
|
||||||
|
</div>
|
||||||
|
{stage.approval.approvals?.length ? (
|
||||||
|
<ul className="datasources-diagnostic-list">
|
||||||
|
{stage.approval.approvals.map((item, index) => (
|
||||||
|
<li key={`${item.actor_ref ?? "actor"}-${index}`}>
|
||||||
|
<span>{item.actor_ref ?? "Unknown actor"} · {readableToken(item.decision ?? "decision")}</span>
|
||||||
|
<small>{item.reason ?? "No reason recorded"} · {formatDate(item.decided_at)}</small>
|
||||||
|
</li>
|
||||||
|
))}
|
||||||
|
</ul>
|
||||||
|
) : <div className="datasources-validation-ok">No approval decision has been recorded.</div>}
|
||||||
|
</ContentSection>
|
||||||
{schemaChanges.length ? (
|
{schemaChanges.length ? (
|
||||||
<ContentSection>
|
<ContentSection>
|
||||||
<ActionToolbar surface="section-header" className="datasources-section-heading">
|
<ActionToolbar surface="section-header" className="datasources-section-heading">
|
||||||
@@ -796,6 +950,211 @@ function StageDetail({ stage }: { stage: DatasourceStage }) {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function StageDecisionDialog({
|
||||||
|
open,
|
||||||
|
settings,
|
||||||
|
stage,
|
||||||
|
onClose,
|
||||||
|
onDecided
|
||||||
|
}: {
|
||||||
|
open: boolean;
|
||||||
|
settings: ApiSettings;
|
||||||
|
stage: DatasourceStage | null;
|
||||||
|
onClose: () => void;
|
||||||
|
onDecided: (stage: DatasourceStage) => void | Promise<void>;
|
||||||
|
}) {
|
||||||
|
const [decision, setDecision] = useState<"approve" | "reject">("approve");
|
||||||
|
const [reason, setReason] = useState("");
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
const [error, setError] = useState("");
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!open) return;
|
||||||
|
setDecision("approve");
|
||||||
|
setReason("");
|
||||||
|
setError("");
|
||||||
|
}, [open, stage?.ref]);
|
||||||
|
|
||||||
|
const submit = async () => {
|
||||||
|
if (!stage?.approval.policy_hash || !stage.approval.subject_digest) return;
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const updated = await decideDatasourceStage(settings, stage.ref, {
|
||||||
|
decision,
|
||||||
|
reason,
|
||||||
|
expected_policy_hash: stage.approval.policy_hash,
|
||||||
|
expected_subject_digest: stage.approval.subject_digest
|
||||||
|
});
|
||||||
|
await onDecided(updated);
|
||||||
|
} catch (operationError) {
|
||||||
|
setError(apiErrorMessage(operationError));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
return (
|
||||||
|
<Dialog
|
||||||
|
open={open}
|
||||||
|
title="Decide datasource promotion"
|
||||||
|
onClose={() => !busy && onClose()}
|
||||||
|
footer={<>
|
||||||
|
<Button onClick={onClose} disabled={busy}>Cancel</Button>
|
||||||
|
<Button
|
||||||
|
variant={decision === "reject" ? "danger" : "primary"}
|
||||||
|
onClick={() => void submit()}
|
||||||
|
disabled={busy || reason.trim().length < 5 || !stage?.approval.policy_hash}
|
||||||
|
>
|
||||||
|
{decision === "approve" ? "Approve stage" : "Reject stage"}
|
||||||
|
</Button>
|
||||||
|
</>}
|
||||||
|
>
|
||||||
|
{error ? <DismissibleAlert tone="danger" resetKey={error}>{error}</DismissibleAlert> : null}
|
||||||
|
<p className="datasources-dialog-copy">
|
||||||
|
The decision is bound to this exact stage, validation result, policy hash, and your account authority.
|
||||||
|
</p>
|
||||||
|
<div className="datasources-key-values">
|
||||||
|
<span><small>Stage</small><strong>{stage?.name ?? "Unavailable"}</strong></span>
|
||||||
|
<span><small>Approval progress</small><strong>{stage?.approval.approval_count ?? 0} / {stage?.approval.required_approvals ?? 1}</strong></span>
|
||||||
|
<span><small>Policy hash</small><strong>{shortFingerprint(stage?.approval.policy_hash ?? "")}</strong></span>
|
||||||
|
<span><small>Expires</small><strong>{formatDate(stage?.approval.expires_at)}</strong></span>
|
||||||
|
</div>
|
||||||
|
<SegmentedControl<"approve" | "reject">
|
||||||
|
ariaLabel="Approval decision"
|
||||||
|
width="fill"
|
||||||
|
size="equal"
|
||||||
|
options={[
|
||||||
|
{ id: "approve", label: "Approve" },
|
||||||
|
{ id: "reject", label: "Reject" }
|
||||||
|
]}
|
||||||
|
value={decision}
|
||||||
|
onChange={setDecision}
|
||||||
|
/>
|
||||||
|
<FormField label="Decision reason" interfaceId="datasources.action.approve" helpContextId="datasources.action.approve" helpModuleId="datasources">
|
||||||
|
<textarea value={reason} onChange={(event) => setReason(event.target.value)} placeholder="Record what was reviewed and why this decision is justified." />
|
||||||
|
</FormField>
|
||||||
|
</Dialog>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function RetentionDialog({
|
||||||
|
open,
|
||||||
|
settings,
|
||||||
|
onClose,
|
||||||
|
onApplied
|
||||||
|
}: {
|
||||||
|
open: boolean;
|
||||||
|
settings: ApiSettings;
|
||||||
|
onClose: () => void;
|
||||||
|
onApplied: (count: number) => void | Promise<void>;
|
||||||
|
}) {
|
||||||
|
const [plan, setPlan] = useState<DatasourceRetentionPlan | null>(null);
|
||||||
|
const [selected, setSelected] = useState<string[]>([]);
|
||||||
|
const [loading, setLoading] = useState(false);
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
const [confirming, setConfirming] = useState(false);
|
||||||
|
const [error, setError] = useState("");
|
||||||
|
|
||||||
|
const loadPlan = useCallback(async () => {
|
||||||
|
setLoading(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const next = await previewDatasourceRetention(settings);
|
||||||
|
setPlan(next);
|
||||||
|
setSelected([]);
|
||||||
|
} catch (loadError) {
|
||||||
|
setError(apiErrorMessage(loadError));
|
||||||
|
} finally {
|
||||||
|
setLoading(false);
|
||||||
|
}
|
||||||
|
}, [settings]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (open) void loadPlan();
|
||||||
|
}, [loadPlan, open]);
|
||||||
|
|
||||||
|
const apply = async () => {
|
||||||
|
if (!plan || !selected.length) return;
|
||||||
|
setBusy(true);
|
||||||
|
setError("");
|
||||||
|
try {
|
||||||
|
const result = await applyDatasourceRetention(settings, plan, selected);
|
||||||
|
setConfirming(false);
|
||||||
|
await onApplied(result.disposed_refs.length);
|
||||||
|
} catch (operationError) {
|
||||||
|
setConfirming(false);
|
||||||
|
setError(apiErrorMessage(operationError));
|
||||||
|
await loadPlan();
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const toggle = (ref: string) => setSelected((current) =>
|
||||||
|
current.includes(ref)
|
||||||
|
? current.filter((item) => item !== ref)
|
||||||
|
: [...current, ref]);
|
||||||
|
|
||||||
|
return <>
|
||||||
|
<Dialog
|
||||||
|
open={open}
|
||||||
|
title="Datasource retention preview"
|
||||||
|
size="wide"
|
||||||
|
onClose={() => !busy && onClose()}
|
||||||
|
footer={<>
|
||||||
|
<Button onClick={() => void loadPlan()} disabled={loading || busy}>Reload preview</Button>
|
||||||
|
<Button onClick={onClose} disabled={busy}>Cancel</Button>
|
||||||
|
<Button variant="danger" onClick={() => setConfirming(true)} disabled={busy || loading || !selected.length}>
|
||||||
|
Apply selected dispositions
|
||||||
|
</Button>
|
||||||
|
</>}
|
||||||
|
>
|
||||||
|
{error ? <DismissibleAlert tone="danger" resetKey={error}>{error}</DismissibleAlert> : null}
|
||||||
|
<p className="datasources-dialog-copy">
|
||||||
|
The preview is read-only. Legal holds, current materializations, pending approvals, and publication evidence remain blocked.
|
||||||
|
</p>
|
||||||
|
{plan ? (
|
||||||
|
<div className="datasources-key-values">
|
||||||
|
<span><small>As of</small><strong>{formatDate(plan.as_of)}</strong></span>
|
||||||
|
<span><small>Plan hash</small><strong>{shortFingerprint(plan.plan_hash)}</strong></span>
|
||||||
|
<span><small>Due targets</small><strong>{plan.candidates.length}</strong></span>
|
||||||
|
<span><small>Selected</small><strong>{selected.length}</strong></span>
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
<LoadingFrame loading={loading}>
|
||||||
|
<div className="datasources-table-scroll">
|
||||||
|
<table>
|
||||||
|
<thead><tr><th className="datasources-retention-select-cell" aria-label="Select" /><th>Target</th><th>Disposition</th><th>Eligible since</th><th>Policy / blockers</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
{plan?.candidates.map((candidate) => (
|
||||||
|
<tr key={candidate.ref}>
|
||||||
|
<td className="datasources-retention-select-cell"><input type="checkbox" aria-label={`Select ${candidate.ref}`} checked={selected.includes(candidate.ref)} disabled={!candidate.eligible} onChange={() => toggle(candidate.ref)} /></td>
|
||||||
|
<td><strong>{candidate.ref}</strong><small>{candidate.datasource_ref ?? "Unpromoted stage"}</small></td>
|
||||||
|
<td>{readableToken(candidate.disposition)}</td>
|
||||||
|
<td>{formatDate(candidate.eligible_at)}</td>
|
||||||
|
<td>{candidate.blockers.length ? candidate.blockers.map(readableToken).join(", ") : `v${candidate.policy_version} · eligible`}</td>
|
||||||
|
</tr>
|
||||||
|
))}
|
||||||
|
{!plan?.candidates.length ? <tr><td colSpan={5}>No retention targets are due.</td></tr> : null}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
</LoadingFrame>
|
||||||
|
</Dialog>
|
||||||
|
<ConfirmDialog
|
||||||
|
open={confirming}
|
||||||
|
title="Apply datasource retention"
|
||||||
|
message={`Dispose ${selected.length} selected target${selected.length === 1 ? "" : "s"}? Payload rows or transient stages will be removed; lifecycle evidence remains.`}
|
||||||
|
confirmLabel="Apply retention"
|
||||||
|
tone="danger"
|
||||||
|
busy={busy}
|
||||||
|
onCancel={() => setConfirming(false)}
|
||||||
|
onConfirm={() => void apply()}
|
||||||
|
/>
|
||||||
|
</>;
|
||||||
|
}
|
||||||
|
|
||||||
function ValidationDiagnosticList({
|
function ValidationDiagnosticList({
|
||||||
diagnostics
|
diagnostics
|
||||||
}: {
|
}: {
|
||||||
@@ -898,6 +1257,9 @@ function GovernanceDialog({
|
|||||||
const [draft, setDraft] = useState<DatasourceGovernance | null>(null);
|
const [draft, setDraft] = useState<DatasourceGovernance | null>(null);
|
||||||
const [freshness, setFreshness] = useState("{}");
|
const [freshness, setFreshness] = useState("{}");
|
||||||
const [quality, setQuality] = useState("{}");
|
const [quality, setQuality] = useState("{}");
|
||||||
|
const [visibility, setVisibility] = useState("{}");
|
||||||
|
const [approval, setApproval] = useState("{}");
|
||||||
|
const [retention, setRetention] = useState("{}");
|
||||||
const [baselineKey, setBaselineKey] = useState("");
|
const [baselineKey, setBaselineKey] = useState("");
|
||||||
const [busy, setBusy] = useState(false);
|
const [busy, setBusy] = useState(false);
|
||||||
const [error, setError] = useState("");
|
const [error, setError] = useState("");
|
||||||
@@ -908,14 +1270,20 @@ function GovernanceDialog({
|
|||||||
const nextDraft = structuredClone(datasource.governance);
|
const nextDraft = structuredClone(datasource.governance);
|
||||||
const nextFreshness = JSON.stringify(datasource.governance.freshness_policy, null, 2);
|
const nextFreshness = JSON.stringify(datasource.governance.freshness_policy, null, 2);
|
||||||
const nextQuality = JSON.stringify(datasource.governance.quality_policy, null, 2);
|
const nextQuality = JSON.stringify(datasource.governance.quality_policy, null, 2);
|
||||||
|
const nextVisibility = JSON.stringify(datasource.governance.visibility_policy ?? {}, null, 2);
|
||||||
|
const nextApproval = JSON.stringify(datasource.governance.approval_policy ?? {}, null, 2);
|
||||||
|
const nextRetention = JSON.stringify(datasource.governance.retention_policy ?? {}, null, 2);
|
||||||
setDraft(nextDraft);
|
setDraft(nextDraft);
|
||||||
setFreshness(nextFreshness);
|
setFreshness(nextFreshness);
|
||||||
setQuality(nextQuality);
|
setQuality(nextQuality);
|
||||||
setBaselineKey(JSON.stringify({ draft: nextDraft, freshness: nextFreshness, quality: nextQuality }));
|
setVisibility(nextVisibility);
|
||||||
|
setApproval(nextApproval);
|
||||||
|
setRetention(nextRetention);
|
||||||
|
setBaselineKey(JSON.stringify({ draft: nextDraft, freshness: nextFreshness, quality: nextQuality, visibility: nextVisibility, approval: nextApproval, retention: nextRetention }));
|
||||||
setError("");
|
setError("");
|
||||||
}, [datasource, open]);
|
}, [datasource, open]);
|
||||||
|
|
||||||
const dirty = Boolean(open && draft && JSON.stringify({ draft, freshness, quality }) !== baselineKey);
|
const dirty = Boolean(open && draft && JSON.stringify({ draft, freshness, quality, visibility, approval, retention }) !== baselineKey);
|
||||||
|
|
||||||
const save = async (): Promise<boolean> => {
|
const save = async (): Promise<boolean> => {
|
||||||
if (!datasource || !draft) return false;
|
if (!datasource || !draft) return false;
|
||||||
@@ -925,7 +1293,10 @@ function GovernanceDialog({
|
|||||||
const updated = await updateDatasourceGovernance(settings, datasource.ref, {
|
const updated = await updateDatasourceGovernance(settings, datasource.ref, {
|
||||||
...draft,
|
...draft,
|
||||||
freshness_policy: parseObject(freshness, "Freshness policy"),
|
freshness_policy: parseObject(freshness, "Freshness policy"),
|
||||||
quality_policy: parseObject(quality, "Quality policy")
|
quality_policy: parseObject(quality, "Quality policy"),
|
||||||
|
visibility_policy: parseObject(visibility, "Visibility policy"),
|
||||||
|
approval_policy: parseObject(approval, "Approval policy"),
|
||||||
|
retention_policy: parseObject(retention, "Retention policy")
|
||||||
});
|
});
|
||||||
await onSaved(updated);
|
await onSaved(updated);
|
||||||
return true;
|
return true;
|
||||||
@@ -1020,6 +1391,9 @@ function GovernanceDialog({
|
|||||||
<FormField label="Retention policy" interfaceId="datasources.field.retention-policy" helpContextId="datasources.field.retention-policy" helpModuleId="datasources" documentation={DATASOURCE_GOVERNANCE_DOCUMENTATION}>
|
<FormField label="Retention policy" interfaceId="datasources.field.retention-policy" helpContextId="datasources.field.retention-policy" helpModuleId="datasources" documentation={DATASOURCE_GOVERNANCE_DOCUMENTATION}>
|
||||||
<input value={draft.retention_policy_ref ?? ""} onChange={(event) => setValue("retention_policy_ref", event.target.value || null)} />
|
<input value={draft.retention_policy_ref ?? ""} onChange={(event) => setValue("retention_policy_ref", event.target.value || null)} />
|
||||||
</FormField>
|
</FormField>
|
||||||
|
<FormField label="Access policy reference" interfaceId="datasources.field.access-policy" helpContextId="datasources.field.access-policy" helpModuleId="datasources" documentation={DATASOURCE_VISIBILITY_DOCUMENTATION}>
|
||||||
|
<input value={draft.access_policy_ref ?? ""} onChange={(event) => setValue("access_policy_ref", event.target.value || null)} placeholder="Optional Policy module target" />
|
||||||
|
</FormField>
|
||||||
<FormField label="Transfer agreement" interfaceId="datasources.field.transfer-agreement" helpContextId="datasources.field.transfer-agreement" helpModuleId="datasources" documentation={DATASOURCE_GOVERNANCE_DOCUMENTATION}>
|
<FormField label="Transfer agreement" interfaceId="datasources.field.transfer-agreement" helpContextId="datasources.field.transfer-agreement" helpModuleId="datasources" documentation={DATASOURCE_GOVERNANCE_DOCUMENTATION}>
|
||||||
<input value={draft.transfer_agreement_ref ?? ""} onChange={(event) => setValue("transfer_agreement_ref", event.target.value || null)} />
|
<input value={draft.transfer_agreement_ref ?? ""} onChange={(event) => setValue("transfer_agreement_ref", event.target.value || null)} />
|
||||||
</FormField>
|
</FormField>
|
||||||
@@ -1034,18 +1408,30 @@ function GovernanceDialog({
|
|||||||
<GovernanceListField label="Purposes" values={draft.purposes} onChange={(values) => setValue("purposes", values)} />
|
<GovernanceListField label="Purposes" values={draft.purposes} onChange={(values) => setValue("purposes", values)} />
|
||||||
<GovernanceListField label="Legal basis references" values={draft.legal_basis_refs} onChange={(values) => setValue("legal_basis_refs", values)} />
|
<GovernanceListField label="Legal basis references" values={draft.legal_basis_refs} onChange={(values) => setValue("legal_basis_refs", values)} />
|
||||||
<GovernanceListField label="Official keys" values={draft.official_keys} onChange={(values) => setValue("official_keys", values)} />
|
<GovernanceListField label="Official keys" values={draft.official_keys} onChange={(values) => setValue("official_keys", values)} />
|
||||||
<GovernanceListField label="Legal hold references" values={draft.hold_refs} onChange={(values) => setValue("hold_refs", values)} />
|
<GovernanceListField label="Legal hold references" helpContextId="datasources.field.retention-policy-contract" values={draft.hold_refs} onChange={(values) => setValue("hold_refs", values)} />
|
||||||
<GovernanceListField label="Affected services and processes" values={draft.affected_refs} onChange={(values) => setValue("affected_refs", values)} />
|
<GovernanceListField label="Affected services and processes" values={draft.affected_refs} onChange={(values) => setValue("affected_refs", values)} />
|
||||||
<GovernanceListField label="Dependent flows, reports, controls and decisions" values={draft.dependency_refs} onChange={(values) => setValue("dependency_refs", values)} />
|
<GovernanceListField label="Dependent flows, reports, controls and decisions" values={draft.dependency_refs} onChange={(values) => setValue("dependency_refs", values)} />
|
||||||
<GovernanceListField label="Known limits" values={draft.known_limits} onChange={(values) => setValue("known_limits", values)} />
|
<GovernanceListField label="Known limits" values={draft.known_limits} onChange={(values) => setValue("known_limits", values)} />
|
||||||
</FormGrid>
|
</FormGrid>
|
||||||
<FormGrid columns={2} gap="small" collapseAt="narrow">
|
<FormGrid columns={2} gap="small" collapseAt="narrow">
|
||||||
|
<FormField label="Visibility policy (JSON)" interfaceId="datasources.field.visibility-policy" helpContextId="datasources.field.visibility-policy" helpModuleId="datasources" documentation={DATASOURCE_VISIBILITY_DOCUMENTATION}>
|
||||||
|
<textarea value={visibility} onChange={(event) => setVisibility(event.target.value)} spellCheck={false} />
|
||||||
|
<small>Configure source and materialization ACLs, field redaction or omission, and principal-bound row filters.</small>
|
||||||
|
</FormField>
|
||||||
<FormField label="Freshness policy (JSON)" documentation={DATASOURCE_GOVERNANCE_DOCUMENTATION}>
|
<FormField label="Freshness policy (JSON)" documentation={DATASOURCE_GOVERNANCE_DOCUMENTATION}>
|
||||||
<textarea value={freshness} onChange={(event) => setFreshness(event.target.value)} spellCheck={false} />
|
<textarea value={freshness} onChange={(event) => setFreshness(event.target.value)} spellCheck={false} />
|
||||||
</FormField>
|
</FormField>
|
||||||
<FormField label="Quality policy (JSON)" documentation={DATASOURCE_GOVERNANCE_DOCUMENTATION}>
|
<FormField label="Quality policy (JSON)" documentation={DATASOURCE_GOVERNANCE_DOCUMENTATION}>
|
||||||
<textarea value={quality} onChange={(event) => setQuality(event.target.value)} spellCheck={false} />
|
<textarea value={quality} onChange={(event) => setQuality(event.target.value)} spellCheck={false} />
|
||||||
</FormField>
|
</FormField>
|
||||||
|
<FormField label="Approval policy (JSON)" interfaceId="datasources.field.approval-policy" helpContextId="datasources.field.approval-policy" helpModuleId="datasources" documentation={DATASOURCE_GOVERNANCE_DOCUMENTATION}>
|
||||||
|
<textarea value={approval} onChange={(event) => setApproval(event.target.value)} spellCheck={false} />
|
||||||
|
<small>Configure a version, required approvals, separation of duties, and optional expiry.</small>
|
||||||
|
</FormField>
|
||||||
|
<FormField label="Retention policy (JSON)" interfaceId="datasources.field.retention-policy-contract" helpContextId="datasources.field.retention-policy-contract" helpModuleId="datasources" documentation={DATASOURCE_GOVERNANCE_DOCUMENTATION}>
|
||||||
|
<textarea value={retention} onChange={(event) => setRetention(event.target.value)} spellCheck={false} />
|
||||||
|
<small>Configure explicit stage, materialization, and frozen-evidence durations; execution always starts with a preview.</small>
|
||||||
|
</FormField>
|
||||||
</FormGrid>
|
</FormGrid>
|
||||||
</DialogSection>
|
</DialogSection>
|
||||||
) : null}
|
) : null}
|
||||||
@@ -1055,15 +1441,17 @@ function GovernanceDialog({
|
|||||||
|
|
||||||
function GovernanceListField({
|
function GovernanceListField({
|
||||||
label,
|
label,
|
||||||
|
helpContextId,
|
||||||
values,
|
values,
|
||||||
onChange
|
onChange
|
||||||
}: {
|
}: {
|
||||||
label: string;
|
label: string;
|
||||||
|
helpContextId?: string;
|
||||||
values: string[];
|
values: string[];
|
||||||
onChange: (values: string[]) => void;
|
onChange: (values: string[]) => void;
|
||||||
}) {
|
}) {
|
||||||
return (
|
return (
|
||||||
<FormField label={label}>
|
<FormField label={label} helpContextId={helpContextId} helpModuleId="datasources">
|
||||||
<textarea
|
<textarea
|
||||||
value={values.join("\n")}
|
value={values.join("\n")}
|
||||||
onChange={(event) => onChange(splitLines(event.target.value))}
|
onChange={(event) => onChange(splitLines(event.target.value))}
|
||||||
@@ -1111,6 +1499,7 @@ function AddDatasourceDialog({
|
|||||||
const [rowsText, setRowsText] = useState('[\n { "id": 1 }\n]');
|
const [rowsText, setRowsText] = useState('[\n { "id": 1 }\n]');
|
||||||
const [csvText, setCsvText] = useState("");
|
const [csvText, setCsvText] = useState("");
|
||||||
const [delimiter, setDelimiter] = useState(";");
|
const [delimiter, setDelimiter] = useState(";");
|
||||||
|
const [csvValueMode, setCsvValueMode] = useState<"text" | "legacy_typed">("text");
|
||||||
const [baselineKey, setBaselineKey] = useState("");
|
const [baselineKey, setBaselineKey] = useState("");
|
||||||
const [busy, setBusy] = useState(false);
|
const [busy, setBusy] = useState(false);
|
||||||
const [error, setError] = useState("");
|
const [error, setError] = useState("");
|
||||||
@@ -1129,6 +1518,7 @@ function AddDatasourceDialog({
|
|||||||
setRowsText('[\n { "id": 1 }\n]');
|
setRowsText('[\n { "id": 1 }\n]');
|
||||||
setCsvText("");
|
setCsvText("");
|
||||||
setDelimiter(";");
|
setDelimiter(";");
|
||||||
|
setCsvValueMode("text");
|
||||||
setBaselineKey(addDatasourceDraftKey({
|
setBaselineKey(addDatasourceDraftKey({
|
||||||
kind: initialKind,
|
kind: initialKind,
|
||||||
format: "csv",
|
format: "csv",
|
||||||
@@ -1140,7 +1530,8 @@ function AddDatasourceDialog({
|
|||||||
description: initialOrigin?.description ?? "",
|
description: initialOrigin?.description ?? "",
|
||||||
rowsText: '[\n { "id": 1 }\n]',
|
rowsText: '[\n { "id": 1 }\n]',
|
||||||
csvText: "",
|
csvText: "",
|
||||||
delimiter: ";"
|
delimiter: ";",
|
||||||
|
csvValueMode: "text"
|
||||||
}));
|
}));
|
||||||
setError("");
|
setError("");
|
||||||
}, [initialKind, initialOrigin, open]);
|
}, [initialKind, initialOrigin, open]);
|
||||||
@@ -1157,7 +1548,8 @@ function AddDatasourceDialog({
|
|||||||
description,
|
description,
|
||||||
rowsText,
|
rowsText,
|
||||||
csvText,
|
csvText,
|
||||||
delimiter
|
delimiter,
|
||||||
|
csvValueMode
|
||||||
}) !== baselineKey);
|
}) !== baselineKey);
|
||||||
|
|
||||||
const chooseOrigin = (ref: string) => {
|
const chooseOrigin = (ref: string) => {
|
||||||
@@ -1209,7 +1601,8 @@ function AddDatasourceDialog({
|
|||||||
...common,
|
...common,
|
||||||
format: "csv",
|
format: "csv",
|
||||||
csv_text: csvText,
|
csv_text: csvText,
|
||||||
delimiter
|
delimiter,
|
||||||
|
csv_value_mode: csvValueMode
|
||||||
})
|
})
|
||||||
: await createDatasourceStage(settings, {
|
: await createDatasourceStage(settings, {
|
||||||
...common,
|
...common,
|
||||||
@@ -1403,6 +1796,16 @@ function AddDatasourceDialog({
|
|||||||
<option value="|">Pipe</option>
|
<option value="|">Pipe</option>
|
||||||
</select>
|
</select>
|
||||||
</FormField>
|
</FormField>
|
||||||
|
<FormField label="CSV values" documentation={DATASOURCE_FIELDS_DOCUMENTATION}>
|
||||||
|
<select
|
||||||
|
value={csvValueMode}
|
||||||
|
onChange={(event) => setCsvValueMode(event.target.value as "text" | "legacy_typed")}
|
||||||
|
disabled={busy}
|
||||||
|
>
|
||||||
|
<option value="text">Preserve text (no automatic conversion)</option>
|
||||||
|
<option value="legacy_typed">Infer types (legacy)</option>
|
||||||
|
</select>
|
||||||
|
</FormField>
|
||||||
<FormField label="CSV data">
|
<FormField label="CSV data">
|
||||||
<textarea
|
<textarea
|
||||||
value={csvText}
|
value={csvText}
|
||||||
@@ -1463,17 +1866,18 @@ function SchemaTable({ fields }: { fields: Datasource["schema"] }) {
|
|||||||
<div className="datasources-table-scroll">
|
<div className="datasources-table-scroll">
|
||||||
<table>
|
<table>
|
||||||
<thead>
|
<thead>
|
||||||
<tr><th>Field</th><th>Type</th><th>Nullable</th></tr>
|
<tr><th>Field</th><th>Type</th><th>Classification</th><th>Nullable</th></tr>
|
||||||
</thead>
|
</thead>
|
||||||
<tbody>
|
<tbody>
|
||||||
{fields.map((field) => (
|
{fields.map((field) => (
|
||||||
<tr key={field.name}>
|
<tr key={field.name}>
|
||||||
<td>{field.name}</td>
|
<td>{field.name}</td>
|
||||||
<td>{field.data_type}</td>
|
<td>{field.data_type}</td>
|
||||||
|
<td>{readableToken(field.classification)}</td>
|
||||||
<td>{field.nullable ? "Yes" : "No"}</td>
|
<td>{field.nullable ? "Yes" : "No"}</td>
|
||||||
</tr>
|
</tr>
|
||||||
))}
|
))}
|
||||||
{!fields.length ? <tr><td colSpan={3}>Schema not available</td></tr> : null}
|
{!fields.length ? <tr><td colSpan={4}>Schema not available</td></tr> : null}
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
</div>
|
</div>
|
||||||
@@ -1520,6 +1924,7 @@ function addDatasourceDraftKey(value: {
|
|||||||
rowsText: string;
|
rowsText: string;
|
||||||
csvText: string;
|
csvText: string;
|
||||||
delimiter: string;
|
delimiter: string;
|
||||||
|
csvValueMode: "text" | "legacy_typed";
|
||||||
}): string {
|
}): string {
|
||||||
return JSON.stringify(value);
|
return JSON.stringify(value);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -15,6 +15,11 @@ export const DATASOURCE_GOVERNANCE_DOCUMENTATION = {
|
|||||||
documentationType: "admin"
|
documentationType: "admin"
|
||||||
} satisfies DocumentationHelpReference;
|
} satisfies DocumentationHelpReference;
|
||||||
|
|
||||||
|
export const DATASOURCE_VISIBILITY_DOCUMENTATION = {
|
||||||
|
topicId: "datasources.visibility",
|
||||||
|
documentationType: "admin"
|
||||||
|
} satisfies DocumentationHelpReference;
|
||||||
|
|
||||||
export const DATASOURCES_I18N = {
|
export const DATASOURCES_I18N = {
|
||||||
loading: "i18n:govoplan-datasources.loading_reason",
|
loading: "i18n:govoplan-datasources.loading_reason",
|
||||||
working: "i18n:govoplan-datasources.working_reason",
|
working: "i18n:govoplan-datasources.working_reason",
|
||||||
|
|||||||
@@ -1,6 +1,10 @@
|
|||||||
import type { PlatformTranslations } from "@govoplan/core-webui";
|
import type { PlatformTranslations } from "@govoplan/core-webui";
|
||||||
|
|
||||||
const en = {
|
const en = {
|
||||||
|
"CSV values": "CSV values",
|
||||||
|
"Preserve text (no automatic conversion)": "Preserve text (no automatic conversion)",
|
||||||
|
"Infer types (legacy)": "Infer types (legacy)",
|
||||||
|
"Loading materializations...": "Loading materializations...",
|
||||||
"i18n:govoplan-datasources.datasources": "Datasources",
|
"i18n:govoplan-datasources.datasources": "Datasources",
|
||||||
"i18n:govoplan-datasources.catalogue": "Datasource catalogue",
|
"i18n:govoplan-datasources.catalogue": "Datasource catalogue",
|
||||||
"i18n:govoplan-datasources.staging": "Datasource staging",
|
"i18n:govoplan-datasources.staging": "Datasource staging",
|
||||||
@@ -63,6 +67,10 @@ const en = {
|
|||||||
"Semantic definition": "Semantic definition",
|
"Semantic definition": "Semantic definition",
|
||||||
"Freshness policy (JSON)": "Freshness policy (JSON)",
|
"Freshness policy (JSON)": "Freshness policy (JSON)",
|
||||||
"Quality policy (JSON)": "Quality policy (JSON)",
|
"Quality policy (JSON)": "Quality policy (JSON)",
|
||||||
|
"Access policy reference": "Access policy reference",
|
||||||
|
"Optional Policy module target": "Optional Policy module target",
|
||||||
|
"Visibility policy (JSON)": "Visibility policy (JSON)",
|
||||||
|
"Configure source and materialization ACLs, field redaction or omission, and principal-bound row filters.": "Configure source and materialization ACLs, field redaction or omission, and principal-bound row filters.",
|
||||||
"Quality policy": "Quality policy",
|
"Quality policy": "Quality policy",
|
||||||
"Policy hash": "Policy hash",
|
"Policy hash": "Policy hash",
|
||||||
"Schema change": "Schema change",
|
"Schema change": "Schema change",
|
||||||
@@ -74,6 +82,10 @@ const en = {
|
|||||||
} as const;
|
} as const;
|
||||||
|
|
||||||
const de: Record<keyof typeof en, string> = {
|
const de: Record<keyof typeof en, string> = {
|
||||||
|
"CSV values": "CSV-Werte",
|
||||||
|
"Preserve text (no automatic conversion)": "Text erhalten (keine automatische Umwandlung)",
|
||||||
|
"Infer types (legacy)": "Typen ableiten (bisheriges Verhalten)",
|
||||||
|
"Loading materializations...": "Materialisierungen werden geladen...",
|
||||||
"i18n:govoplan-datasources.datasources": "Datenquellen",
|
"i18n:govoplan-datasources.datasources": "Datenquellen",
|
||||||
"i18n:govoplan-datasources.catalogue": "Datenquellenkatalog",
|
"i18n:govoplan-datasources.catalogue": "Datenquellenkatalog",
|
||||||
"i18n:govoplan-datasources.staging": "Datenquellen-Staging",
|
"i18n:govoplan-datasources.staging": "Datenquellen-Staging",
|
||||||
@@ -136,6 +148,10 @@ const de: Record<keyof typeof en, string> = {
|
|||||||
"Semantic definition": "Semantische Definition",
|
"Semantic definition": "Semantische Definition",
|
||||||
"Freshness policy (JSON)": "Aktualitätsrichtlinie (JSON)",
|
"Freshness policy (JSON)": "Aktualitätsrichtlinie (JSON)",
|
||||||
"Quality policy (JSON)": "Qualitätsrichtlinie (JSON)",
|
"Quality policy (JSON)": "Qualitätsrichtlinie (JSON)",
|
||||||
|
"Access policy reference": "Zugriffsrichtlinienreferenz",
|
||||||
|
"Optional Policy module target": "Optionales Ziel im Richtlinienmodul",
|
||||||
|
"Visibility policy (JSON)": "Sichtbarkeitsrichtlinie (JSON)",
|
||||||
|
"Configure source and materialization ACLs, field redaction or omission, and principal-bound row filters.": "Konfigurieren Sie Zugriffslisten für Quellen und Materialisierungen, Feldschwärzung oder -auslassung sowie an den Akteur gebundene Zeilenfilter.",
|
||||||
"Quality policy": "Qualitätsrichtlinie",
|
"Quality policy": "Qualitätsrichtlinie",
|
||||||
"Policy hash": "Richtlinien-Hash",
|
"Policy hash": "Richtlinien-Hash",
|
||||||
"Schema change": "Schemaänderung",
|
"Schema change": "Schemaänderung",
|
||||||
|
|||||||
+1
-1
@@ -10,7 +10,7 @@ const readScopes = ["datasources:catalogue:read", "datasources:source:admin"];
|
|||||||
export const datasourcesModule: PlatformWebModule = {
|
export const datasourcesModule: PlatformWebModule = {
|
||||||
id: "datasources",
|
id: "datasources",
|
||||||
label: "i18n:govoplan-datasources.datasources",
|
label: "i18n:govoplan-datasources.datasources",
|
||||||
version: "0.1.18",
|
version: "0.1.22",
|
||||||
optionalDependencies: [
|
optionalDependencies: [
|
||||||
"access",
|
"access",
|
||||||
"audit",
|
"audit",
|
||||||
|
|||||||
@@ -287,6 +287,17 @@
|
|||||||
max-height: min(860px, calc(100vh - 32px));
|
max-height: min(860px, calc(100vh - 32px));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.datasources-retention-select-cell {
|
||||||
|
width: 2.5rem;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
.datasources-table-scroll td small {
|
||||||
|
display: block;
|
||||||
|
margin-top: 0.2rem;
|
||||||
|
color: var(--color-text-muted);
|
||||||
|
}
|
||||||
|
|
||||||
.datasources-governance-dialog .datasources-dialog-fields textarea {
|
.datasources-governance-dialog .datasources-dialog-fields textarea {
|
||||||
min-height: 76px;
|
min-height: 76px;
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user