Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4d570b5e9e | ||
|
|
a6c5bab3a4 | ||
|
|
4175262b8b | ||
|
|
618f10fe89 | ||
|
|
f222be63b2 | ||
|
|
2c97223fb4 | ||
|
|
5b989a7f6c | ||
|
|
08c1ecbf81 | ||
|
|
c1111c605f | ||
|
|
a86220db27 | ||
|
|
6767905cbb | ||
|
|
d12e0bce7d | ||
|
|
21da2e1ad4 | ||
|
|
20d134898a | ||
|
|
f908ddbf9d | ||
|
|
b14c693bde | ||
|
|
3dae80b6b8 |
@@ -19,6 +19,21 @@ without storing the previewed row contents.
|
||||
- **Risk Compliance:** sanctions matching policy, review, dispositions, and
|
||||
legal evidence.
|
||||
|
||||
## Data-subject requests
|
||||
|
||||
Dataflow publishes `privacy.dsar.dataflow` for exact pipeline, revision,
|
||||
reconciliation, run, deployment, trigger, and delivery references and for
|
||||
minimized operator or automation-authority attribution. It never exports
|
||||
graphs, SQL, request/event payloads, reconciliation corrections, authorization
|
||||
snapshots, provenance bodies, errors, source details, hashes, credentials, or
|
||||
output rows. Authoritative input modules locate and correct subject facts;
|
||||
Dataflow does not guess identity by scanning arbitrary transformations.
|
||||
|
||||
Exact terminal run and delivery detail can be minimized idempotently, and
|
||||
subject-linked automation authority can be disabled and revoked. Definitions,
|
||||
active work, decisions, deployments, broad pipeline packages, published
|
||||
Datasource outputs, and institutional attribution require review or retention.
|
||||
|
||||
## Node Library
|
||||
|
||||
The canonical backend catalogue is exposed to the WebUI and groups executable
|
||||
@@ -122,6 +137,13 @@ records the effective Policy decision and ancestor limits. Inherited
|
||||
definitions remain read-only; lower scopes may narrow, but not broaden,
|
||||
execution, reuse, inheritance, or automation permissions.
|
||||
|
||||
Derived definitions report when their source has a newer immutable revision;
|
||||
the source never mutates the child silently. Adopting an update requires the
|
||||
reviewed source revision and hash plus a reason. It appends a new child
|
||||
revision, retains the previous graph and all run evidence, records reviewer
|
||||
and Policy provenance, and returns the child to draft before the changed graph
|
||||
can run or receive automation.
|
||||
|
||||
Complete active flows support explicit user/API starts, administrative
|
||||
backfills, one-time schedules, interval schedules, and exact-match platform
|
||||
events. Trigger deliveries are durable and idempotent. They enqueue the same
|
||||
@@ -132,11 +154,15 @@ the run before source access or output publication.
|
||||
Confidential and restricted events are not accepted through the direct
|
||||
ingress; those require Core's transactional event bridge.
|
||||
|
||||
Reusable subflow nodes pin a template reference, version, graph snapshot, and
|
||||
parameter values. Their single input is bound to an explicitly marked inline
|
||||
source inside the snapshot, parameter substitution is data-only, and nesting
|
||||
is bounded. This keeps completed run definitions reproducible even when the
|
||||
source template changes later.
|
||||
Reusable subflow nodes select a Policy-authorized complete flow or template and
|
||||
an immutable revision. The server resolves the graph instead of accepting a
|
||||
caller-supplied snapshot, records the source hash and Policy decision, and pins
|
||||
closed typed input/output contracts. Their single input is bound to an
|
||||
explicitly marked typed inline source inside the snapshot, parameter
|
||||
substitution is data-only, and cycles across nested references are rejected.
|
||||
Incompatible caller schemas fail validation before execution. This keeps
|
||||
completed run definitions reproducible even when the source definition changes
|
||||
later.
|
||||
|
||||
The executable fixtures in `fixtures/golden` cover monthly structured-file
|
||||
reconciliation, sanctions screening, a HEICO-style current-status export, and
|
||||
@@ -152,6 +178,16 @@ logical row disappeared. It never silently applies a correction to business
|
||||
data; a downstream governed transform or Workflow handoff must interpret the
|
||||
recorded action.
|
||||
|
||||
For saved reconciliation pipelines, the preview results provide a review
|
||||
dialog for those rows. Reviewers create a tenant-owned decision set and append
|
||||
accept, reject, correct, or defer decisions with a mandatory reason. Writes use
|
||||
optimistic concurrency; updating a decision creates another immutable revision
|
||||
rather than replacing history. The current projection appears in the ordinary
|
||||
Dataflow source catalogue as `dataflow-decision-set:<id>` and carries a content
|
||||
fingerprint. A changed input hash is therefore shown as stale and cannot be
|
||||
silently reused. Corrections remain annotations until an explicit downstream
|
||||
transform applies them.
|
||||
|
||||
Reporting consumers may either evaluate a pinned pipeline revision or pin one
|
||||
successful published run. An exact run pin is immutable: it cannot be supplied
|
||||
new parameters, and Dataflow reads only the recorded Datasource materialization
|
||||
@@ -177,3 +213,21 @@ npm run test:structure
|
||||
|
||||
The implementation epic is
|
||||
[`govoplan-dataflow#1`](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/1).
|
||||
|
||||
## Git-source WebUI package
|
||||
|
||||
The repository root exposes `@govoplan/dataflow-webui` for Git-tagged release
|
||||
dependencies. It mirrors the owning `webui/package.json` version, public
|
||||
TypeScript/CSS exports and peer requirements, with entry paths under
|
||||
`webui/src`. Consumers provide the shared Core/React peers; the facade runs no
|
||||
development or install scripts. The source archive contains `webui/src`, this
|
||||
README and any repository license file. Run module development checks from `webui/`; Python
|
||||
installation remains governed by `pyproject.toml`.
|
||||
|
||||
Das Repository stellt `@govoplan/dataflow-webui` am Wurzelpfad für versionierte
|
||||
Git-Abhängigkeiten bereit. Version, öffentliche TypeScript-/CSS-Exporte und
|
||||
Peer-Anforderungen entsprechen `webui/package.json`; die Einstiegspfade liegen
|
||||
unter `webui/src`. Gemeinsame Core-/React-Peers stellt die einbindende Anwendung
|
||||
bereit. Die Fassade führt keine Entwicklungs- oder Installationsskripte aus.
|
||||
Entwicklungsprüfungen bleiben in `webui/`, die Python-Installation weiterhin in
|
||||
`pyproject.toml` definiert.
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"name": "@govoplan/dataflow-webui",
|
||||
"version": "0.1.25",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "webui/src/index.ts",
|
||||
"module": "webui/src/index.ts",
|
||||
"types": "webui/src/index.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./webui/src/index.ts",
|
||||
"import": "./webui/src/index.ts"
|
||||
},
|
||||
"./styles/dataflow.css": "./webui/src/styles/dataflow.css"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@govoplan/core-webui": "^0.1.46",
|
||||
"@xyflow/react": "^12.11.2",
|
||||
"lucide-react": "^1.23.0",
|
||||
"react": ">=19.2.7 <20",
|
||||
"react-dom": ">=19.2.7 <20",
|
||||
"react-router": ">=8.3.0 <9",
|
||||
"typescript": "^5.7.2"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@govoplan/core-webui": {
|
||||
"optional": true
|
||||
}
|
||||
},
|
||||
"files": [
|
||||
"webui/src",
|
||||
"README.md",
|
||||
"LICENSE"
|
||||
]
|
||||
}
|
||||
+2
-2
@@ -4,14 +4,14 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "govoplan-dataflow"
|
||||
version = "0.1.17"
|
||||
version = "0.1.25"
|
||||
description = "Governed graphical and SQL data pipelines for GovOPlaN."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
license = "AGPL-3.0-or-later"
|
||||
authors = [{ name = "GovOPlaN" }]
|
||||
dependencies = [
|
||||
"govoplan-core>=0.1.17",
|
||||
"govoplan-core>=0.1.46",
|
||||
"sqlglot>=30.14,<31",
|
||||
]
|
||||
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
from __future__ import annotations
|
||||
|
||||
__version__ = "0.1.17"
|
||||
__version__ = "0.1.25"
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
import math
|
||||
from typing import Any, Mapping, Protocol, runtime_checkable
|
||||
|
||||
from govoplan_dataflow.backend.batches import TypedBatch
|
||||
@@ -33,8 +34,8 @@ class ExecutionBudget:
|
||||
raise ValueError("Execution output row limit must be positive.")
|
||||
if self.max_batch_bytes < 1:
|
||||
raise ValueError("Execution byte limit must be positive.")
|
||||
if self.max_wall_seconds <= 0:
|
||||
raise ValueError("Execution time limit must be positive.")
|
||||
if not math.isfinite(self.max_wall_seconds) or self.max_wall_seconds <= 0:
|
||||
raise ValueError("Execution time limit must be finite and positive.")
|
||||
if self.max_memory_bytes < 64 * 1024 * 1024:
|
||||
raise ValueError("Execution memory limit must be at least 64 MiB.")
|
||||
if self.max_concurrency < 1:
|
||||
@@ -106,11 +107,19 @@ class BackendExecutionError(RuntimeError):
|
||||
code: str = "backend.execution",
|
||||
node_id: str | None = None,
|
||||
diagnostics: tuple[DataflowDiagnostic, ...] = (),
|
||||
node_diagnostics: tuple[NodePreviewDiagnostic, ...] = (),
|
||||
source_fingerprints: tuple[dict[str, Any], ...] = (),
|
||||
input_row_count: int = 0,
|
||||
node_preview: NodePreviewResult | None = None,
|
||||
) -> None:
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
self.node_id = node_id
|
||||
self.diagnostics = diagnostics
|
||||
self.node_diagnostics = node_diagnostics
|
||||
self.source_fingerprints = source_fingerprints
|
||||
self.input_row_count = input_row_count
|
||||
self.node_preview = node_preview
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
|
||||
@@ -1,10 +1,25 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import asdict
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
|
||||
from govoplan_core.security.bounded_process import (
|
||||
ProcessBudgetError,
|
||||
ProcessLimits,
|
||||
run_bounded_operation,
|
||||
)
|
||||
from govoplan_core.security.worker_payload import (
|
||||
decode_worker_payload,
|
||||
encode_worker_payload,
|
||||
)
|
||||
from govoplan_dataflow.backend.backends.base import (
|
||||
BackendExecutionError,
|
||||
BackendExecutionRequest,
|
||||
BackendExecutionResult,
|
||||
BackendSource,
|
||||
ExecutionBudget,
|
||||
canonical_result_schema,
|
||||
)
|
||||
from govoplan_dataflow.backend.batches import TypedBatch
|
||||
@@ -14,8 +29,17 @@ from govoplan_dataflow.backend.executor import (
|
||||
ResolvedSource,
|
||||
execute_preview,
|
||||
)
|
||||
from govoplan_dataflow.backend.ir import IrExecutionResult, ir_to_graph
|
||||
from govoplan_dataflow.backend.schemas import GraphNode
|
||||
from govoplan_dataflow.backend.ir import IrExecutionResult, IrSchema, TypedGraphIr, ir_to_graph
|
||||
from govoplan_dataflow.backend.planner import ExecutionPlan
|
||||
from govoplan_dataflow.backend.schemas import (
|
||||
DataflowDiagnostic,
|
||||
GraphNode,
|
||||
NodePreviewDiagnostic,
|
||||
NodePreviewResult,
|
||||
)
|
||||
|
||||
|
||||
_TRANSPORT_BYTES = 32 * 1024 * 1024
|
||||
|
||||
|
||||
class ReferenceExecutionBackend:
|
||||
@@ -32,6 +56,52 @@ class ReferenceExecutionBackend:
|
||||
self,
|
||||
request: BackendExecutionRequest,
|
||||
) -> BackendExecutionResult:
|
||||
_validate_source_batches(request)
|
||||
try:
|
||||
limits = ProcessLimits(
|
||||
wall_seconds=request.budget.max_wall_seconds,
|
||||
cpu_seconds=max(1, math.ceil(request.budget.max_wall_seconds)),
|
||||
memory_bytes=request.budget.max_memory_bytes,
|
||||
input_bytes=_TRANSPORT_BYTES,
|
||||
output_bytes=_TRANSPORT_BYTES,
|
||||
)
|
||||
payload = encode_worker_payload(_request_payload(request), max_bytes=_TRANSPORT_BYTES)
|
||||
response = decode_worker_payload(
|
||||
run_bounded_operation(_execute_reference_worker, payload, limits=limits),
|
||||
max_bytes=_TRANSPORT_BYTES,
|
||||
)
|
||||
except ProcessBudgetError as exc:
|
||||
raise BackendExecutionError(str(exc), code=f"backend.process.{exc.code}") from exc
|
||||
except ValueError as exc:
|
||||
raise BackendExecutionError(
|
||||
"Reference execution exceeds its supported process/transport budget.",
|
||||
code="backend.budget",
|
||||
) from exc
|
||||
if "error" in response:
|
||||
error = response["error"]
|
||||
raise BackendExecutionError(
|
||||
error["message"], code=error["code"], node_id=error["node_id"],
|
||||
diagnostics=tuple(DataflowDiagnostic.model_validate(item) for item in error["diagnostics"]),
|
||||
node_diagnostics=tuple(NodePreviewDiagnostic.model_validate(item) for item in error["node_diagnostics"]),
|
||||
source_fingerprints=error["source_fingerprints"],
|
||||
input_row_count=error["input_row_count"],
|
||||
node_preview=NodePreviewResult.model_validate(error["node_preview"]) if error["node_preview"] else None,
|
||||
)
|
||||
batch = _batch_from_payload(response["batch"])
|
||||
batch.ensure_within(
|
||||
max_rows=request.budget.max_output_rows,
|
||||
max_bytes=request.budget.max_batch_bytes,
|
||||
)
|
||||
return BackendExecutionResult(
|
||||
contract=IrExecutionResult.model_validate(response["contract"]),
|
||||
batch=batch,
|
||||
node_diagnostics=tuple(NodePreviewDiagnostic.model_validate(item) for item in response["node_diagnostics"]),
|
||||
node_preview=NodePreviewResult.model_validate(response["node_preview"]) if response["node_preview"] else None,
|
||||
metadata=response["metadata"],
|
||||
)
|
||||
|
||||
def _execute_in_process(self, request: BackendExecutionRequest) -> BackendExecutionResult:
|
||||
"""Pure reference evaluation, called only inside the disposable worker."""
|
||||
_validate_source_batches(request)
|
||||
try:
|
||||
result = execute_preview(
|
||||
@@ -50,6 +120,10 @@ class ReferenceExecutionBackend:
|
||||
code="backend.reference",
|
||||
node_id=exc.node_id,
|
||||
diagnostics=tuple(exc.diagnostics),
|
||||
node_diagnostics=tuple(exc.node_diagnostics),
|
||||
source_fingerprints=tuple(exc.source_fingerprints),
|
||||
input_row_count=exc.input_row_count,
|
||||
node_preview=exc.node_preview,
|
||||
) from exc
|
||||
observed_batch = TypedBatch.from_rows(result.rows)
|
||||
batch = TypedBatch.from_rows(
|
||||
@@ -97,6 +171,90 @@ class ReferenceExecutionBackend:
|
||||
)
|
||||
|
||||
|
||||
def _batch_payload(batch: TypedBatch) -> dict:
|
||||
return {
|
||||
"schema": batch.schema.model_dump(mode="python"),
|
||||
"columns": dict(batch.columns),
|
||||
"row_count": batch.row_count,
|
||||
"byte_count": batch.byte_count,
|
||||
}
|
||||
|
||||
|
||||
def _batch_from_payload(value: dict) -> TypedBatch:
|
||||
return TypedBatch(
|
||||
schema=IrSchema.model_validate(value["schema"]), columns=value["columns"],
|
||||
row_count=value["row_count"], byte_count=value["byte_count"],
|
||||
)
|
||||
|
||||
|
||||
def _request_payload(request: BackendExecutionRequest) -> dict:
|
||||
plan = request.plan
|
||||
return {
|
||||
"plan": {
|
||||
"graph": plan.graph.model_dump(mode="python"),
|
||||
"ordered_node_ids": plan.ordered_node_ids,
|
||||
"diagnostics": tuple(item.model_dump(mode="python") for item in plan.diagnostics),
|
||||
"generated_sql": plan.generated_sql,
|
||||
"sql_diagnostics": tuple(item.model_dump(mode="python") for item in plan.sql_diagnostics),
|
||||
"semantic_hash": plan.semantic_hash,
|
||||
},
|
||||
"budget": asdict(request.budget),
|
||||
"preview_node_id": request.preview_node_id,
|
||||
"sources": {
|
||||
key: {
|
||||
"node_id": source.node_id, "batch": _batch_payload(source.batch),
|
||||
"source_ref": source.source_ref, "provider": source.provider,
|
||||
"fingerprint": source.fingerprint, "total_rows": source.total_rows,
|
||||
"truncated": source.truncated, "source_name": source.source_name,
|
||||
"kind": source.kind,
|
||||
}
|
||||
for key, source in request.sources.items()
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _execute_reference_worker(payload: bytes) -> bytes:
|
||||
"""Data-only worker boundary; no database, provider callback or principal."""
|
||||
value = decode_worker_payload(payload, max_bytes=_TRANSPORT_BYTES)
|
||||
plan = value["plan"]
|
||||
request = BackendExecutionRequest(
|
||||
plan=ExecutionPlan(
|
||||
graph=TypedGraphIr.model_validate(plan["graph"]),
|
||||
ordered_node_ids=plan["ordered_node_ids"],
|
||||
diagnostics=tuple(DataflowDiagnostic.model_validate(item) for item in plan["diagnostics"]),
|
||||
generated_sql=plan["generated_sql"],
|
||||
sql_diagnostics=tuple(DataflowDiagnostic.model_validate(item) for item in plan["sql_diagnostics"]),
|
||||
semantic_hash=plan["semantic_hash"],
|
||||
),
|
||||
budget=ExecutionBudget(**value["budget"]),
|
||||
preview_node_id=value["preview_node_id"],
|
||||
sources={
|
||||
key: BackendSource(**{**source, "batch": _batch_from_payload(source["batch"])})
|
||||
for key, source in value["sources"].items()
|
||||
},
|
||||
)
|
||||
try:
|
||||
result = ReferenceExecutionBackend()._execute_in_process(request)
|
||||
except BackendExecutionError as exc:
|
||||
response = {"error": {
|
||||
"message": str(exc), "code": exc.code, "node_id": exc.node_id,
|
||||
"diagnostics": tuple(item.model_dump(mode="python") for item in exc.diagnostics),
|
||||
"node_diagnostics": tuple(item.model_dump(mode="python") for item in exc.node_diagnostics),
|
||||
"source_fingerprints": exc.source_fingerprints,
|
||||
"input_row_count": exc.input_row_count,
|
||||
"node_preview": exc.node_preview.model_dump(mode="python") if exc.node_preview else None,
|
||||
}}
|
||||
else:
|
||||
response = {
|
||||
"contract": result.contract.model_dump(mode="python"),
|
||||
"batch": _batch_payload(result.batch),
|
||||
"node_diagnostics": tuple(item.model_dump(mode="python") for item in result.node_diagnostics),
|
||||
"node_preview": result.node_preview.model_dump(mode="python") if result.node_preview else None,
|
||||
"metadata": dict(result.metadata),
|
||||
}
|
||||
return encode_worker_payload(response, max_bytes=_TRANSPORT_BYTES)
|
||||
|
||||
|
||||
def _validate_source_batches(request: BackendExecutionRequest) -> None:
|
||||
for source in request.sources.values():
|
||||
try:
|
||||
@@ -139,6 +297,7 @@ def _source_for_node(
|
||||
node: GraphNode,
|
||||
) -> BackendSource | None:
|
||||
candidates = (
|
||||
reference_source_key(node),
|
||||
node.id,
|
||||
str(node.config.get("source_ref") or ""),
|
||||
str(node.config.get("source_name") or ""),
|
||||
@@ -153,4 +312,10 @@ def _source_for_node(
|
||||
)
|
||||
|
||||
|
||||
def reference_source_key(node: GraphNode) -> str:
|
||||
"""Nested graphs may reuse node IDs; bind resolved data to the full config."""
|
||||
content = json.dumps(node.config, sort_keys=True, separators=(",", ":"), default=str)
|
||||
return "reference-config:" + hashlib.sha256(content.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
__all__ = ["ReferenceExecutionBackend"]
|
||||
|
||||
@@ -17,6 +17,7 @@ from sqlalchemy import (
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from govoplan_core.core.concurrency import strong_resource_etag
|
||||
from govoplan_core.db.base import Base, TimestampMixin
|
||||
|
||||
|
||||
@@ -121,6 +122,11 @@ class DataflowPipeline(Base, TimestampMixin):
|
||||
cascade="all, delete-orphan",
|
||||
order_by="DataflowTrigger.created_at",
|
||||
)
|
||||
decision_sets: Mapped[list["DataflowReconciliationDecisionSet"]] = relationship(
|
||||
back_populates="pipeline",
|
||||
cascade="all, delete-orphan",
|
||||
order_by="DataflowReconciliationDecisionSet.name",
|
||||
)
|
||||
|
||||
|
||||
class DataflowPipelineRevision(Base, TimestampMixin):
|
||||
@@ -153,6 +159,99 @@ class DataflowPipelineRevision(Base, TimestampMixin):
|
||||
pipeline: Mapped[DataflowPipeline] = relationship(back_populates="revisions")
|
||||
|
||||
|
||||
class DataflowReconciliationDecisionSet(Base, TimestampMixin):
|
||||
__tablename__ = "dataflow_reconciliation_decision_sets"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"pipeline_id",
|
||||
"name",
|
||||
name="uq_dataflow_decision_sets_pipeline_name",
|
||||
),
|
||||
Index(
|
||||
"ix_dataflow_decision_sets_tenant_pipeline",
|
||||
"tenant_id",
|
||||
"pipeline_id",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
pipeline_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("dataflow_pipelines.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
name: Mapped[str] = mapped_column(String(300), nullable=False)
|
||||
node_id: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||
resource_revision: Mapped[int] = mapped_column(
|
||||
Integer,
|
||||
default=1,
|
||||
nullable=False,
|
||||
)
|
||||
created_by: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
updated_by: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
|
||||
pipeline: Mapped[DataflowPipeline] = relationship(back_populates="decision_sets")
|
||||
decisions: Mapped[list["DataflowReconciliationDecision"]] = relationship(
|
||||
back_populates="decision_set",
|
||||
cascade="all, delete-orphan",
|
||||
order_by="DataflowReconciliationDecision.revision",
|
||||
)
|
||||
|
||||
@property
|
||||
def strong_etag(self) -> str:
|
||||
return strong_resource_etag(
|
||||
"dataflow_reconciliation_decision_set",
|
||||
self.id,
|
||||
self.resource_revision,
|
||||
)
|
||||
|
||||
|
||||
class DataflowReconciliationDecision(Base, TimestampMixin):
|
||||
__tablename__ = "dataflow_reconciliation_decisions"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"decision_set_id",
|
||||
"revision",
|
||||
name="uq_dataflow_reconciliation_decision_revision",
|
||||
),
|
||||
Index(
|
||||
"ix_dataflow_reconciliation_decisions_current",
|
||||
"tenant_id",
|
||||
"decision_set_id",
|
||||
"key_hash",
|
||||
"revision",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
decision_set_id: Mapped[str] = mapped_column(
|
||||
ForeignKey(
|
||||
"dataflow_reconciliation_decision_sets.id",
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
revision: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
key_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
input_hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
action: Mapped[str] = mapped_column(String(20), nullable=False)
|
||||
reason: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
correction: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
|
||||
actor_ref: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
decided_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
decision_set: Mapped[DataflowReconciliationDecisionSet] = relationship(
|
||||
back_populates="decisions"
|
||||
)
|
||||
|
||||
|
||||
class DataflowRun(Base, TimestampMixin):
|
||||
__tablename__ = "dataflow_runs"
|
||||
__table_args__ = (
|
||||
|
||||
@@ -0,0 +1,835 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.dsar import (
|
||||
DsarErasureActionRef,
|
||||
DsarExecutionResultRef,
|
||||
DsarRecordRef,
|
||||
DsarSubjectRef,
|
||||
dsar_capability_name,
|
||||
)
|
||||
from govoplan_dataflow.backend.db.models import (
|
||||
DataflowPipeline,
|
||||
DataflowPipelineDeployment,
|
||||
DataflowPipelineRevision,
|
||||
DataflowReconciliationDecision,
|
||||
DataflowReconciliationDecisionSet,
|
||||
DataflowRun,
|
||||
DataflowTrigger,
|
||||
DataflowTriggerDelivery,
|
||||
)
|
||||
|
||||
|
||||
DATAFLOW_DSAR_CAPABILITY = dsar_capability_name("dataflow")
|
||||
_MAX_RECORDS = 5_000
|
||||
_CONFLICT = object()
|
||||
_DIRECT_ALIASES = {
|
||||
"pipeline_id": ("dataflow.pipeline",),
|
||||
"revision_id": ("dataflow.pipeline_revision", "dataflow.revision"),
|
||||
"decision_set_id": ("dataflow.decision_set",),
|
||||
"decision_id": ("dataflow.decision",),
|
||||
"run_id": ("dataflow.run",),
|
||||
"deployment_id": ("dataflow.deployment",),
|
||||
"trigger_id": ("dataflow.trigger",),
|
||||
"delivery_id": ("dataflow.trigger_delivery", "dataflow.delivery"),
|
||||
}
|
||||
_RESOURCE_MODELS = {
|
||||
"dataflow_pipeline": DataflowPipeline,
|
||||
"dataflow_pipeline_revision": DataflowPipelineRevision,
|
||||
"dataflow_decision_set": DataflowReconciliationDecisionSet,
|
||||
"dataflow_decision": DataflowReconciliationDecision,
|
||||
"dataflow_run": DataflowRun,
|
||||
"dataflow_deployment": DataflowPipelineDeployment,
|
||||
"dataflow_trigger": DataflowTrigger,
|
||||
"dataflow_trigger_delivery": DataflowTriggerDelivery,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _Selectors:
|
||||
account_id: str | None
|
||||
identity_id: str | None
|
||||
membership_id: str | None
|
||||
direct: dict[str, str]
|
||||
|
||||
@property
|
||||
def actor_ids(self) -> tuple[str, ...]:
|
||||
return tuple(
|
||||
value
|
||||
for value in (self.account_id, self.identity_id, self.membership_id)
|
||||
if value
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _Match:
|
||||
resource_type: str
|
||||
row: Any
|
||||
category: str
|
||||
|
||||
|
||||
class DataflowDsarProvider:
|
||||
provider_id = "dataflow"
|
||||
module_id = "dataflow"
|
||||
|
||||
def search_subject(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
) -> Sequence[DsarRecordRef]:
|
||||
db = _session(session)
|
||||
selectors = _selectors(subject)
|
||||
if selectors is None or not (selectors.actor_ids or selectors.direct):
|
||||
return ()
|
||||
direct = _direct_matches(db, tenant_id=tenant_id, selectors=selectors)
|
||||
if direct is None:
|
||||
return ()
|
||||
if direct:
|
||||
if selectors.actor_ids and not all(
|
||||
_correlates(
|
||||
db,
|
||||
tenant_id=tenant_id,
|
||||
match=match,
|
||||
actor_ids=selectors.actor_ids,
|
||||
)
|
||||
for match in direct
|
||||
):
|
||||
return ()
|
||||
matches = direct
|
||||
else:
|
||||
matches = _canonical_matches(
|
||||
db,
|
||||
tenant_id=tenant_id,
|
||||
actor_ids=selectors.actor_ids,
|
||||
)
|
||||
|
||||
records: list[DsarRecordRef] = []
|
||||
seen: set[tuple[str, str]] = set()
|
||||
for match in matches:
|
||||
key = (match.resource_type, str(match.row.id))
|
||||
if key in seen:
|
||||
continue
|
||||
if len(records) >= _MAX_RECORDS:
|
||||
raise ValueError(
|
||||
"Dataflow DSAR result limit exceeded; narrow the selectors."
|
||||
)
|
||||
seen.add(key)
|
||||
records.append(_record(match))
|
||||
return tuple(records)
|
||||
|
||||
def plan_erasure(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
records: Sequence[DsarRecordRef],
|
||||
) -> Sequence[DsarErasureActionRef]:
|
||||
del tenant_id
|
||||
_session(session)
|
||||
if _selectors(subject) is None:
|
||||
raise ValueError("Dataflow DSAR subject selectors conflict.")
|
||||
actions: list[DsarErasureActionRef] = []
|
||||
for record in records:
|
||||
_validate_record(record)
|
||||
kind = _planned_kind(record)
|
||||
executable = kind in {"anonymize", "revoke"}
|
||||
actions.append(
|
||||
DsarErasureActionRef(
|
||||
action_id=(
|
||||
f"dataflow:{kind}:{record.resource_type}:{record.resource_id}"
|
||||
),
|
||||
provider_id=self.provider_id,
|
||||
module_id=self.module_id,
|
||||
kind=kind,
|
||||
resource_type=record.resource_type,
|
||||
resource_id=record.resource_id,
|
||||
title=(
|
||||
f"Minimize {record.title}"
|
||||
if kind == "anonymize"
|
||||
else f"{kind.replace('_', ' ').title()} {record.title}"
|
||||
),
|
||||
rationale=_rationale(record, kind=kind),
|
||||
executable=executable,
|
||||
irreversible=kind == "anonymize",
|
||||
metadata={"record_category": record.category},
|
||||
)
|
||||
)
|
||||
return tuple(actions)
|
||||
|
||||
def execute_erasure(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
actions: Sequence[DsarErasureActionRef],
|
||||
request_id: str,
|
||||
) -> Sequence[DsarExecutionResultRef]:
|
||||
db = _session(session)
|
||||
selectors = _selectors(subject)
|
||||
if selectors is None:
|
||||
raise ValueError("Dataflow DSAR subject selectors conflict.")
|
||||
results: list[DsarExecutionResultRef] = []
|
||||
for action in actions:
|
||||
_validate_action(action)
|
||||
if not action.executable:
|
||||
results.append(
|
||||
DsarExecutionResultRef(
|
||||
action_id=action.action_id,
|
||||
status="blocked",
|
||||
summary=(
|
||||
"Review pipeline dependencies, published outputs, "
|
||||
"retention, and institutional evidence."
|
||||
),
|
||||
evidence={"request_id": request_id},
|
||||
)
|
||||
)
|
||||
continue
|
||||
model = _RESOURCE_MODELS[action.resource_type]
|
||||
row = (
|
||||
db.query(model)
|
||||
.filter(model.tenant_id == tenant_id, model.id == action.resource_id)
|
||||
.with_for_update()
|
||||
.one_or_none()
|
||||
)
|
||||
if row is None:
|
||||
status = "unchanged"
|
||||
summary = "Dataflow row was already absent or minimized."
|
||||
else:
|
||||
match = _Match(action.resource_type, row, "execution")
|
||||
if not (
|
||||
_directly_targets(selectors, match)
|
||||
or _correlates(
|
||||
db,
|
||||
tenant_id=tenant_id,
|
||||
match=match,
|
||||
actor_ids=selectors.actor_ids,
|
||||
)
|
||||
or _already_revoked(action.resource_type, row)
|
||||
):
|
||||
raise ValueError(
|
||||
"Dataflow DSAR action is not corroborated by the subject."
|
||||
)
|
||||
status, summary = _execute_action(
|
||||
db,
|
||||
resource_type=action.resource_type,
|
||||
row=row,
|
||||
kind=action.kind,
|
||||
)
|
||||
results.append(
|
||||
DsarExecutionResultRef(
|
||||
action_id=action.action_id,
|
||||
status=status,
|
||||
summary=summary,
|
||||
evidence={"request_id": request_id},
|
||||
)
|
||||
)
|
||||
return tuple(results)
|
||||
|
||||
|
||||
def _direct_matches(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
selectors: _Selectors,
|
||||
) -> list[_Match] | None:
|
||||
matches: list[_Match] = []
|
||||
for selector, raw_value in selectors.direct.items():
|
||||
value = _strip_prefix(raw_value)
|
||||
if selector == "pipeline_id":
|
||||
pipeline = _one(session, DataflowPipeline, tenant_id, value)
|
||||
if pipeline is None:
|
||||
return None
|
||||
current = _pipeline_package(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
pipeline=pipeline,
|
||||
)
|
||||
else:
|
||||
model, resource_type = {
|
||||
"revision_id": (
|
||||
DataflowPipelineRevision,
|
||||
"dataflow_pipeline_revision",
|
||||
),
|
||||
"decision_set_id": (
|
||||
DataflowReconciliationDecisionSet,
|
||||
"dataflow_decision_set",
|
||||
),
|
||||
"decision_id": (
|
||||
DataflowReconciliationDecision,
|
||||
"dataflow_decision",
|
||||
),
|
||||
"run_id": (DataflowRun, "dataflow_run"),
|
||||
"deployment_id": (
|
||||
DataflowPipelineDeployment,
|
||||
"dataflow_deployment",
|
||||
),
|
||||
"trigger_id": (DataflowTrigger, "dataflow_trigger"),
|
||||
"delivery_id": (
|
||||
DataflowTriggerDelivery,
|
||||
"dataflow_trigger_delivery",
|
||||
),
|
||||
}[selector]
|
||||
row = _one(session, model, tenant_id, value)
|
||||
if row is None:
|
||||
return None
|
||||
current = [_Match(resource_type, row, _direct_category(resource_type, row))]
|
||||
matches.extend(current)
|
||||
if len(matches) > _MAX_RECORDS:
|
||||
raise ValueError(
|
||||
"Dataflow DSAR result limit exceeded; narrow the selectors."
|
||||
)
|
||||
return matches
|
||||
|
||||
|
||||
def _pipeline_package(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
pipeline: DataflowPipeline,
|
||||
) -> list[_Match]:
|
||||
matches = [_Match("dataflow_pipeline", pipeline, "dataflow_configuration")]
|
||||
specs = (
|
||||
(DataflowPipelineRevision, "dataflow_pipeline_revision"),
|
||||
(DataflowReconciliationDecisionSet, "dataflow_decision_set"),
|
||||
(DataflowRun, "dataflow_run"),
|
||||
(DataflowPipelineDeployment, "dataflow_deployment"),
|
||||
(DataflowTrigger, "dataflow_trigger"),
|
||||
(DataflowTriggerDelivery, "dataflow_trigger_delivery"),
|
||||
)
|
||||
decision_sets: list[DataflowReconciliationDecisionSet] = []
|
||||
for model, resource_type in specs:
|
||||
rows = (
|
||||
session.query(model)
|
||||
.filter(model.tenant_id == tenant_id, model.pipeline_id == pipeline.id)
|
||||
.order_by(model.id)
|
||||
.limit(_MAX_RECORDS + 1)
|
||||
.all()
|
||||
)
|
||||
if model is DataflowReconciliationDecisionSet:
|
||||
decision_sets = rows
|
||||
matches.extend(
|
||||
_Match(resource_type, row, "pipeline_related_state") for row in rows
|
||||
)
|
||||
decision_set_ids = {row.id for row in decision_sets}
|
||||
if decision_set_ids:
|
||||
decisions = (
|
||||
session.query(DataflowReconciliationDecision)
|
||||
.filter(
|
||||
DataflowReconciliationDecision.tenant_id == tenant_id,
|
||||
DataflowReconciliationDecision.decision_set_id.in_(decision_set_ids),
|
||||
)
|
||||
.order_by(DataflowReconciliationDecision.id)
|
||||
.limit(_MAX_RECORDS + 1)
|
||||
.all()
|
||||
)
|
||||
matches.extend(
|
||||
_Match("dataflow_decision", row, "pipeline_related_state")
|
||||
for row in decisions
|
||||
)
|
||||
return matches
|
||||
|
||||
|
||||
def _canonical_matches(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
actor_ids: tuple[str, ...],
|
||||
) -> list[_Match]:
|
||||
if not actor_ids:
|
||||
return []
|
||||
triggers = (
|
||||
session.query(DataflowTrigger)
|
||||
.filter(
|
||||
DataflowTrigger.tenant_id == tenant_id,
|
||||
or_(
|
||||
DataflowTrigger.authorization_account_id.in_(actor_ids),
|
||||
DataflowTrigger.authorization_membership_id.in_(actor_ids),
|
||||
),
|
||||
)
|
||||
.order_by(DataflowTrigger.id)
|
||||
.limit(_MAX_RECORDS + 1)
|
||||
.all()
|
||||
)
|
||||
matches = [
|
||||
_Match("dataflow_trigger", row, "dataflow_automation_authority")
|
||||
for row in triggers
|
||||
]
|
||||
specs = (
|
||||
(
|
||||
DataflowPipeline,
|
||||
or_(
|
||||
DataflowPipeline.created_by.in_(actor_ids),
|
||||
DataflowPipeline.updated_by.in_(actor_ids),
|
||||
),
|
||||
"dataflow_pipeline",
|
||||
),
|
||||
(
|
||||
DataflowPipelineRevision,
|
||||
DataflowPipelineRevision.created_by.in_(actor_ids),
|
||||
"dataflow_pipeline_revision",
|
||||
),
|
||||
(
|
||||
DataflowReconciliationDecisionSet,
|
||||
or_(
|
||||
DataflowReconciliationDecisionSet.created_by.in_(actor_ids),
|
||||
DataflowReconciliationDecisionSet.updated_by.in_(actor_ids),
|
||||
),
|
||||
"dataflow_decision_set",
|
||||
),
|
||||
(
|
||||
DataflowReconciliationDecision,
|
||||
DataflowReconciliationDecision.actor_ref.in_(actor_ids),
|
||||
"dataflow_decision",
|
||||
),
|
||||
(DataflowRun, DataflowRun.created_by.in_(actor_ids), "dataflow_run"),
|
||||
(
|
||||
DataflowPipelineDeployment,
|
||||
DataflowPipelineDeployment.promoted_by.in_(actor_ids),
|
||||
"dataflow_deployment",
|
||||
),
|
||||
(
|
||||
DataflowTrigger,
|
||||
or_(
|
||||
DataflowTrigger.created_by.in_(actor_ids),
|
||||
DataflowTrigger.updated_by.in_(actor_ids),
|
||||
),
|
||||
"dataflow_trigger",
|
||||
),
|
||||
)
|
||||
for model, condition, resource_type in specs:
|
||||
rows = (
|
||||
session.query(model)
|
||||
.filter(model.tenant_id == tenant_id, condition)
|
||||
.order_by(model.id)
|
||||
.limit(_MAX_RECORDS + 1)
|
||||
.all()
|
||||
)
|
||||
matches.extend(
|
||||
_Match(resource_type, row, "dataflow_operator_attribution") for row in rows
|
||||
)
|
||||
if len(matches) > _MAX_RECORDS:
|
||||
raise ValueError(
|
||||
"Dataflow DSAR result limit exceeded; narrow the selectors."
|
||||
)
|
||||
return matches
|
||||
|
||||
|
||||
def _one(session: Session, model: Any, tenant_id: str, row_id: str) -> Any | None:
|
||||
return (
|
||||
session.query(model)
|
||||
.filter(model.tenant_id == tenant_id, model.id == row_id)
|
||||
.one_or_none()
|
||||
)
|
||||
|
||||
|
||||
def _direct_category(resource_type: str, row: Any) -> str:
|
||||
if resource_type == "dataflow_run":
|
||||
return (
|
||||
"terminal_dataflow_run"
|
||||
if row.status in {"succeeded", "failed", "cancelled", "outcome_unknown"}
|
||||
else "active_dataflow_run"
|
||||
)
|
||||
if resource_type == "dataflow_trigger_delivery":
|
||||
return (
|
||||
"terminal_trigger_delivery"
|
||||
if row.status in {"succeeded", "failed", "skipped", "cancelled"}
|
||||
else "active_trigger_delivery"
|
||||
)
|
||||
return "dataflow_configuration"
|
||||
|
||||
|
||||
def _correlates(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
match: _Match,
|
||||
actor_ids: tuple[str, ...],
|
||||
) -> bool:
|
||||
row = match.row
|
||||
if any(
|
||||
str(getattr(row, field, "") or "") in actor_ids
|
||||
for field in (
|
||||
"created_by",
|
||||
"updated_by",
|
||||
"promoted_by",
|
||||
"actor_ref",
|
||||
"authorization_account_id",
|
||||
"authorization_membership_id",
|
||||
)
|
||||
):
|
||||
return True
|
||||
if match.resource_type == "dataflow_trigger_delivery":
|
||||
trigger = session.get(DataflowTrigger, row.trigger_id)
|
||||
if (
|
||||
trigger
|
||||
and trigger.tenant_id == tenant_id
|
||||
and any(
|
||||
str(getattr(trigger, field, "") or "") in actor_ids
|
||||
for field in (
|
||||
"created_by",
|
||||
"updated_by",
|
||||
"authorization_account_id",
|
||||
"authorization_membership_id",
|
||||
)
|
||||
)
|
||||
):
|
||||
return True
|
||||
pipeline_id = getattr(row, "pipeline_id", None)
|
||||
if not pipeline_id and match.resource_type == "dataflow_decision":
|
||||
decision_set = session.get(
|
||||
DataflowReconciliationDecisionSet,
|
||||
row.decision_set_id,
|
||||
)
|
||||
pipeline_id = decision_set.pipeline_id if decision_set else None
|
||||
if pipeline_id:
|
||||
pipeline = session.get(DataflowPipeline, pipeline_id)
|
||||
return bool(
|
||||
pipeline
|
||||
and pipeline.tenant_id == tenant_id
|
||||
and (pipeline.created_by in actor_ids or pipeline.updated_by in actor_ids)
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
def _directly_targets(selectors: _Selectors, match: _Match) -> bool:
|
||||
selector = {
|
||||
"dataflow_pipeline": "pipeline_id",
|
||||
"dataflow_pipeline_revision": "revision_id",
|
||||
"dataflow_decision_set": "decision_set_id",
|
||||
"dataflow_decision": "decision_id",
|
||||
"dataflow_run": "run_id",
|
||||
"dataflow_deployment": "deployment_id",
|
||||
"dataflow_trigger": "trigger_id",
|
||||
"dataflow_trigger_delivery": "delivery_id",
|
||||
}[match.resource_type]
|
||||
if _strip_prefix(selectors.direct.get(selector, "")) == str(match.row.id):
|
||||
return True
|
||||
pipeline_id = _strip_prefix(selectors.direct.get("pipeline_id", ""))
|
||||
return bool(
|
||||
pipeline_id
|
||||
and (
|
||||
(match.resource_type == "dataflow_pipeline" and match.row.id == pipeline_id)
|
||||
or getattr(match.row, "pipeline_id", None) == pipeline_id
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _record(match: _Match) -> DsarRecordRef:
|
||||
row = match.row
|
||||
immutable = match.category == "dataflow_operator_attribution"
|
||||
return DsarRecordRef(
|
||||
provider_id="dataflow",
|
||||
module_id="dataflow",
|
||||
resource_type=match.resource_type,
|
||||
resource_id=str(row.id),
|
||||
category=match.category,
|
||||
title=match.resource_type.removeprefix("dataflow_").replace("_", " ").title(),
|
||||
data={
|
||||
key: value
|
||||
for key, value in _record_data(match.resource_type, row).items()
|
||||
if value is not None
|
||||
},
|
||||
observed_at=_observed_at(row),
|
||||
immutable_evidence=immutable,
|
||||
retention_reason=(
|
||||
"Institutional Dataflow authorship, decisions, runs, and deployment "
|
||||
"activity remain attributable for governance and audit."
|
||||
if immutable
|
||||
else None
|
||||
),
|
||||
source_path="/dataflow",
|
||||
)
|
||||
|
||||
|
||||
def _record_data(resource_type: str, row: Any) -> dict[str, object]:
|
||||
if resource_type == "dataflow_pipeline":
|
||||
return {
|
||||
"scope_type": row.scope_type,
|
||||
"definition_kind": row.definition_kind,
|
||||
"status": row.status,
|
||||
"current_revision": row.current_revision,
|
||||
"allow_run": row.allow_run,
|
||||
"allow_reuse": row.allow_reuse,
|
||||
"allow_automation": row.allow_automation,
|
||||
"deleted_at": _iso(row.deleted_at),
|
||||
"created_at": _iso(row.created_at),
|
||||
"updated_at": _iso(row.updated_at),
|
||||
}
|
||||
if resource_type == "dataflow_pipeline_revision":
|
||||
return {
|
||||
"revision": row.revision,
|
||||
"schema_version": row.schema_version,
|
||||
"editor_mode": row.editor_mode,
|
||||
"created_at": _iso(row.created_at),
|
||||
}
|
||||
if resource_type == "dataflow_decision_set":
|
||||
return {
|
||||
"resource_revision": row.resource_revision,
|
||||
"created_at": _iso(row.created_at),
|
||||
"updated_at": _iso(row.updated_at),
|
||||
}
|
||||
if resource_type == "dataflow_decision":
|
||||
return {
|
||||
"revision": row.revision,
|
||||
"action": row.action,
|
||||
"decided_at": _iso(row.decided_at),
|
||||
}
|
||||
if resource_type == "dataflow_run":
|
||||
return {
|
||||
"run_type": row.run_type,
|
||||
"status": row.status,
|
||||
"execution_backend": row.execution_backend,
|
||||
"environment": row.environment,
|
||||
"invocation_kind": row.invocation_kind,
|
||||
"input_row_count": row.input_row_count,
|
||||
"output_row_count": row.output_row_count,
|
||||
"attempts": row.attempts,
|
||||
"progress_percent": row.progress_percent,
|
||||
"purged_at": _iso(row.purged_at),
|
||||
"started_at": _iso(row.started_at),
|
||||
"finished_at": _iso(row.finished_at),
|
||||
"created_at": _iso(row.created_at),
|
||||
}
|
||||
if resource_type == "dataflow_deployment":
|
||||
return {
|
||||
"environment": row.environment,
|
||||
"source_environment": row.source_environment,
|
||||
"status": row.status,
|
||||
"created_at": _iso(row.created_at),
|
||||
"updated_at": _iso(row.updated_at),
|
||||
}
|
||||
if resource_type == "dataflow_trigger":
|
||||
return {
|
||||
"kind": row.kind,
|
||||
"status": row.status,
|
||||
"revision": row.revision,
|
||||
"catch_up_policy": row.catch_up_policy,
|
||||
"max_concurrent_runs": row.max_concurrent_runs,
|
||||
"next_fire_at": _iso(row.next_fire_at),
|
||||
"last_fire_at": _iso(row.last_fire_at),
|
||||
"created_at": _iso(row.created_at),
|
||||
"updated_at": _iso(row.updated_at),
|
||||
}
|
||||
return {
|
||||
"invocation_kind": row.invocation_kind,
|
||||
"status": row.status,
|
||||
"attempts": row.attempts,
|
||||
"scheduled_for": _iso(row.scheduled_for),
|
||||
"finished_at": _iso(row.finished_at),
|
||||
"created_at": _iso(row.created_at),
|
||||
}
|
||||
|
||||
|
||||
def _planned_kind(record: DsarRecordRef) -> str:
|
||||
if record.category in {"terminal_dataflow_run", "terminal_trigger_delivery"}:
|
||||
return "anonymize"
|
||||
if record.category == "dataflow_automation_authority":
|
||||
return "revoke"
|
||||
if record.category == "dataflow_operator_attribution":
|
||||
return "retain"
|
||||
return "manual_review"
|
||||
|
||||
|
||||
def _rationale(record: DsarRecordRef, *, kind: str) -> str:
|
||||
if kind == "anonymize":
|
||||
return (
|
||||
"Clear retained request, event, authorization, diagnostic, and error "
|
||||
"detail while preserving minimal run or delivery evidence."
|
||||
)
|
||||
if kind == "revoke":
|
||||
return (
|
||||
"Disable automation and remove the subject-linked authorization "
|
||||
"snapshot without deleting historical delivery evidence."
|
||||
)
|
||||
if kind == "retain":
|
||||
return record.retention_reason or "Retain institutional attribution evidence."
|
||||
return (
|
||||
"A Dataflow owner must review definitions, reconciliation decisions, "
|
||||
"active work, published outputs, and downstream dependencies."
|
||||
)
|
||||
|
||||
|
||||
def _execute_action(
|
||||
session: Session,
|
||||
*,
|
||||
resource_type: str,
|
||||
row: Any,
|
||||
kind: str,
|
||||
) -> tuple[str, str]:
|
||||
if resource_type == "dataflow_run" and kind == "anonymize":
|
||||
if row.status not in {"succeeded", "failed", "cancelled", "outcome_unknown"}:
|
||||
raise ValueError("Active Dataflow runs require manual review.")
|
||||
authorization = dict(row.authorization_ or {})
|
||||
authorization.pop("submitted_principal", None)
|
||||
authorization["personal_data_purged"] = True
|
||||
changed = _replace_fields(
|
||||
row,
|
||||
{
|
||||
"request_": {},
|
||||
"source_fingerprints": [],
|
||||
"diagnostics": [],
|
||||
"authorization_": authorization,
|
||||
"correlation_id": None,
|
||||
"causation_id": None,
|
||||
"error": None,
|
||||
},
|
||||
)
|
||||
if row.purged_at is None:
|
||||
row.purged_at = datetime.now(timezone.utc)
|
||||
changed = True
|
||||
elif resource_type == "dataflow_trigger_delivery" and kind == "anonymize":
|
||||
if row.status not in {"succeeded", "failed", "skipped", "cancelled"}:
|
||||
raise ValueError("Active Dataflow deliveries require manual review.")
|
||||
changed = _replace_fields(
|
||||
row,
|
||||
{
|
||||
"event_": None,
|
||||
"authorization_provenance": {},
|
||||
"error": None,
|
||||
},
|
||||
)
|
||||
elif resource_type == "dataflow_trigger" and kind == "revoke":
|
||||
changed = _replace_fields(
|
||||
row,
|
||||
{
|
||||
"status": "disabled",
|
||||
"authorization_account_id": "redacted",
|
||||
"authorization_membership_id": "redacted",
|
||||
"authorization_ref": f"redacted:{row.id}",
|
||||
"grant_scopes": [],
|
||||
"config_": {},
|
||||
"publication_": None,
|
||||
"last_error": None,
|
||||
},
|
||||
)
|
||||
else:
|
||||
raise ValueError("Dataflow DSAR executable action is unsupported.")
|
||||
if changed:
|
||||
session.flush()
|
||||
return "executed", "Personal Dataflow detail minimized."
|
||||
return "unchanged", "Personal Dataflow detail was already minimized."
|
||||
|
||||
|
||||
def _replace_fields(row: Any, values: dict[str, object]) -> bool:
|
||||
changed = False
|
||||
for field, value in values.items():
|
||||
if getattr(row, field) != value:
|
||||
setattr(row, field, value)
|
||||
changed = True
|
||||
return changed
|
||||
|
||||
|
||||
def _already_revoked(resource_type: str, row: Any) -> bool:
|
||||
return bool(
|
||||
resource_type == "dataflow_trigger"
|
||||
and row.status == "disabled"
|
||||
and row.authorization_account_id == "redacted"
|
||||
and row.authorization_membership_id == "redacted"
|
||||
and row.authorization_ref == f"redacted:{row.id}"
|
||||
)
|
||||
|
||||
|
||||
def _selectors(subject: DsarSubjectRef) -> _Selectors | None:
|
||||
refs = subject.external_references
|
||||
canonical = (
|
||||
_coalesce(
|
||||
subject.account_id,
|
||||
refs.get("dataflow.account"),
|
||||
refs.get("access.account"),
|
||||
),
|
||||
_coalesce(
|
||||
subject.identity_id,
|
||||
refs.get("dataflow.identity"),
|
||||
refs.get("identity.id"),
|
||||
),
|
||||
_coalesce(
|
||||
subject.membership_id,
|
||||
refs.get("dataflow.membership"),
|
||||
refs.get("tenancy.membership"),
|
||||
),
|
||||
)
|
||||
if any(value is _CONFLICT for value in canonical):
|
||||
return None
|
||||
direct: dict[str, str] = {}
|
||||
for selector, aliases in _DIRECT_ALIASES.items():
|
||||
value = _coalesce(*(refs.get(alias) for alias in aliases))
|
||||
if value is _CONFLICT:
|
||||
return None
|
||||
if value:
|
||||
direct[selector] = str(value)
|
||||
return _Selectors(
|
||||
account_id=_optional(canonical[0]),
|
||||
identity_id=_optional(canonical[1]),
|
||||
membership_id=_optional(canonical[2]),
|
||||
direct=direct,
|
||||
)
|
||||
|
||||
|
||||
def _coalesce(*values: str | None) -> str | None | object:
|
||||
normalized = {str(value).strip() for value in values if str(value or "").strip()}
|
||||
if len(normalized) > 1:
|
||||
return _CONFLICT
|
||||
return next(iter(normalized), None)
|
||||
|
||||
|
||||
def _optional(value: object) -> str | None:
|
||||
return value if isinstance(value, str) and value else None
|
||||
|
||||
|
||||
def _strip_prefix(value: str) -> str:
|
||||
return value.partition(":")[2] if ":" in value else value
|
||||
|
||||
|
||||
def _observed_at(row: Any) -> datetime | None:
|
||||
for field in ("decided_at", "finished_at", "updated_at", "created_at"):
|
||||
value = getattr(row, field, None)
|
||||
if isinstance(value, datetime):
|
||||
return _aware(value)
|
||||
return None
|
||||
|
||||
|
||||
def _iso(value: datetime | None) -> str | None:
|
||||
aware = _aware(value)
|
||||
return aware.isoformat() if aware else None
|
||||
|
||||
|
||||
def _aware(value: datetime | None) -> datetime | None:
|
||||
if value is None or value.tzinfo is not None:
|
||||
return value
|
||||
return value.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def _session(value: object) -> Session:
|
||||
if not isinstance(value, Session):
|
||||
raise TypeError("Dataflow DSAR requires a SQLAlchemy Session.")
|
||||
return value
|
||||
|
||||
|
||||
def _validate_record(record: DsarRecordRef) -> None:
|
||||
if record.provider_id != "dataflow" or record.module_id != "dataflow":
|
||||
raise ValueError("Dataflow DSAR cannot plan a foreign provider record.")
|
||||
if record.resource_type not in _RESOURCE_MODELS or not record.resource_id:
|
||||
raise ValueError("Dataflow DSAR record identity is invalid.")
|
||||
|
||||
|
||||
def _validate_action(action: DsarErasureActionRef) -> None:
|
||||
if action.provider_id != "dataflow" or action.module_id != "dataflow":
|
||||
raise ValueError("Dataflow DSAR cannot execute a foreign provider action.")
|
||||
if action.resource_type not in _RESOURCE_MODELS or not action.action_id.startswith(
|
||||
"dataflow:"
|
||||
):
|
||||
raise ValueError("Dataflow DSAR action identity is invalid.")
|
||||
|
||||
|
||||
__all__ = ["DATAFLOW_DSAR_CAPABILITY", "DataflowDsarProvider"]
|
||||
@@ -19,6 +19,7 @@ from govoplan_dataflow.backend.operator_registry import (
|
||||
OperatorExecutionContext,
|
||||
OperatorExecutionResult,
|
||||
)
|
||||
from govoplan_dataflow.backend.preview_limits import MAX_RESULT_BYTES
|
||||
from govoplan_dataflow.backend.schemas import (
|
||||
DataflowDiagnostic,
|
||||
GraphNode,
|
||||
@@ -32,7 +33,6 @@ from govoplan_dataflow.backend.subflows import substitute_parameters
|
||||
|
||||
EXECUTOR_VERSION = "dataflow-preview-v2"
|
||||
MAX_EXECUTION_SECONDS = 2.0
|
||||
MAX_RESULT_BYTES = 1_000_000
|
||||
MAX_SOURCE_ROWS = 250
|
||||
MAX_INTERMEDIATE_ROWS = 10_000
|
||||
|
||||
|
||||
@@ -12,6 +12,8 @@ import sqlglot
|
||||
from sqlglot import exp
|
||||
from sqlglot.errors import ParseError
|
||||
|
||||
from govoplan_dataflow.backend.preview_limits import MAX_RESULT_BYTES
|
||||
|
||||
|
||||
ExpressionDataType = Literal[
|
||||
"unknown",
|
||||
@@ -438,6 +440,14 @@ def _evaluate_pad(expression: exp.Expression, row: dict[str, Any]) -> str | None
|
||||
target_length = int(_evaluate(expression.expression, row))
|
||||
if target_length < 0:
|
||||
raise ValueError("Padding length cannot be negative.")
|
||||
# Every character takes at least one serialized byte. Enforce the existing
|
||||
# node budget before padding allocates memory, including when an outer
|
||||
# LENGTH/SUBSTRING would otherwise conceal the oversized intermediate value.
|
||||
# The node's final byte check still accounts for Unicode and JSON overhead.
|
||||
if target_length > MAX_RESULT_BYTES:
|
||||
raise ExpressionError(
|
||||
f"Padding length exceeds the {MAX_RESULT_BYTES:,}-byte preview result limit."
|
||||
)
|
||||
source = str(value)
|
||||
if len(source) >= target_length:
|
||||
return source[:target_length]
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Iterable
|
||||
|
||||
from govoplan_core.core.modules import DocumentationTopic, localize_documentation_topics as _localize_topics
|
||||
|
||||
|
||||
_TRANSLATIONS = {
|
||||
"dataflow.data-subject-requests": {
|
||||
"title": "Datenschutzanfragen zu Dataflow-Daten",
|
||||
"summary": "Gespeicherte Transformationsdetails minimieren, ohne abgeleitete Datenflüsse als führende Betroffenendaten zu behandeln.",
|
||||
"body": (
|
||||
"Dataflow gleicht exakte mandantenbezogene Pipeline-, Revisions-, Abgleich-, Lauf-, Bereitstellungs-, Trigger- und Zustellkennungen sowie minimierte Konto-, Identitäts- und Mitgliedschaftszuordnungen ab. Ergebnisse enthalten niemals Graphen, SQL, Anfrage- oder Ereignisinhalte, Abgleichkorrekturen, Autorisierungssnapshots, Provenienztexte, Fehler, Quelldetails, Hashwerte, Zugangsdaten oder Ausgabezeilen. Dataflow durchsucht beliebige Transformationsinhalte nicht nach Personen; das führende Eingabemodul muss Betroffenendaten auffinden und berichtigen. Eindeutig bestimmte abgeschlossene Lauf- und Zustelldetails können idempotent minimiert und personenbezogene Automatisierungsbefugnisse deaktiviert und widerrufen werden. Definitionen, Abgleichnachweise, aktive Arbeiten, Bereitstellungen, umfassende Pipeline-Pakete, veröffentlichte Datasource-Ausgaben und institutionelle Zuordnungen benötigen eine autorisierte Prüfung oder Aufbewahrung. Nach der Prüfung sind die Quellen zu berichtigen und Ableitungen in Datasources, Search und Reporting zu aktualisieren."
|
||||
),
|
||||
},
|
||||
"dataflow.module-boundary": {
|
||||
"title": "Modulgrenze von Dataflow",
|
||||
"summary": "Versionierte tabellarische Transformationen grafisch oder mit eingeschränktem SQL bearbeiten.",
|
||||
"body": (
|
||||
"Dataflow verantwortet kanonische Pipeline-Graphen, unveränderliche Revisionen, Validierung, eingeschränkte SQL-Kompilierung, Vorschau- und Laufdiagnosen sowie Herkunftsreferenzen. Datasources verantwortet den gesteuerten Katalog und Materialisierungen, Connectors den externen Abruf und Zugangsdaten, Reporting die analytische Darstellung und Exporte, Workflow die Orchestrierung und menschliche Übergaben und Risk Compliance die Sanktionsprüfung und Richtliniengrenzen. Benutzer-SQL wird in freigegebene Transformationen übersetzt und nie ungeprüft an eine Datenbank weitergereicht. "
|
||||
"Das Öffnen oder Neuladen von Dataflow startet keine Pipeline. Kann der Editor nach einer Entwicklungsaktualisierung nicht geladen werden, sichern Sie ungespeicherte Arbeit vor dem Neuladen des Browsers. Administratoren sollten Fehler beim Laden von Oberflächendateien von Fehlern der Pipeline-API oder Zugriffsfehlern unterscheiden. Entwicklungs- und Browser-Konformitätsserver verwenden getrennte Abhängigkeitscaches; ein älterer Server muss nach dieser Konfigurationsaktualisierung gegebenenfalls neu gestartet werden."
|
||||
),
|
||||
},
|
||||
"dataflow.reference.nodes-and-expressions": {
|
||||
"title": "Dataflow-Knoten und Ausdrücke",
|
||||
"summary": "Typisierte Knoteneingaben, Ausdrücke, Schemafortschreibung und begrenzte Zwischenergebnisse verstehen.",
|
||||
"body": (
|
||||
"Jeder Graphknoten definiert typisierte Eingaben, Konfiguration, Ausgabeschema und Validierungsregeln. Quellknoten binden Inline-Inhalte oder gesteuerte Datasource-Referenzen; Verknüpfungs-, Filter-, Transformations-, Qualitäts-, Abgleich-, Teilfluss- und Ausgabeknoten bleiben im kanonischen Graphen ausdrücklich sichtbar. Wiederverwendbare Teilflüsse wählen eine durch Policy erlaubte unveränderliche Fluss- oder Vorlagenrevision. Der Server löst Graph, Quell-Hash, Policy-Entscheidung und geschlossene Ein-/Ausgabeverträge auf und bindet sie; mitgelieferte Graphkopien werden ignoriert, unvereinbare Eingaben und zyklische Referenzen abgelehnt. Abgleichzeilen führen stabile Schlüssel- und Eingabe-Hashes sowie Vorher-/Nachher-Werte. Prüfentscheidungen werden als unveränderliche, mandanteneigene Entscheidungssätze gespeichert; geänderte Eingaben werden ungültig, ohne Fachdaten still umzuschreiben. Ausdrücke führen weder Host- noch Datenbankcode aus. Knoten-Vorschauen sind begrenzt, für die handelnde Person datenschutzgefiltert und werden nicht als Laufergebnis gespeichert. SQL wird in denselben Graphen kompiliert; nicht unterstützte Anweisungen erscheinen als Diagnose. "
|
||||
"Referenz-Vorschauen behalten die bestehende Grenze von 1.000.000 Byte je serialisiertem Knotenergebnis. "
|
||||
"LPAD und RPAD weisen Ziellängen über 1.000.000 Zeichen vor dem Reservieren des Auffüllspeichers zurück. "
|
||||
"Dies gilt auch für übergroße Zwischenergebnisse innerhalb von LENGTH oder SUBSTRING, selbst wenn der endgültige Einzelwert klein wäre. "
|
||||
"Verringern Sie die gewünschte Auffülllänge; der Ausdruck scheitert mit einer Diagnose am betreffenden Knoten, statt Daten abzuschneiden. "
|
||||
"Gewöhnliches Unicode-Auffüllen, NULL-Eingaben und Kürzungen innerhalb der Grenze behalten ihr Verhalten. "
|
||||
"Die abschließende Byteprüfung berücksichtigt weiterhin JSON- und Mehrbyte-Zeichenaufwand. "
|
||||
"Diese Speicherprüfung ersetzt keine Laufzeit- oder Bytegrenzen für andere Ausdrucksoperationen."
|
||||
),
|
||||
},
|
||||
"dataflow.reference.fields-and-consequences": {
|
||||
"title": "Dataflow-Felder und Lebenszyklusfolgen",
|
||||
"summary": "Bedeutung von Geltungsbereich, Revision, Wiederverwendung, Automatisierung, Ausführung, Veröffentlichung, Promotion und Löschung.",
|
||||
"body": (
|
||||
"Der Geltungsbereich bestimmt Eigentum und Policy-Vererbung. Vorlagen können abgeleitet, aber nicht ausgeführt werden; vollständige Flüsse dürfen bei wirksamer Policy geprüft, versioniert, automatisiert und ausgeführt werden. Speichern fügt eine unveränderliche Revision an. Eine bereichsbezogene Kopie bindet Quellrevision und Inhalts-Hash; eine neuere Quelle ändert sie nicht automatisch. Die geprüfte Übernahme benötigt den exakten Quell-Hash und eine Begründung, fügt eine Kopierevision an, protokolliert Policy-Entscheidung und prüfende Person und setzt die Kopie zur erneuten Aktivierung auf Entwurf. Trigger binden Revision und Autorisierungsnachweis und prüfen ihre Befugnis je Zustellung neu. „Läufe zulassen“ ist nur die Zulassungsgrenze der Definition und gewährt niemandem eine Berechtigung. Einmalige Läufe verwenden lokale Mandantenzeit; versäumte Intervalle werden je Richtlinie zu einem Lauf zusammengefasst oder übersprungen, niemals ungeprüft vollständig nachgeholt. Das Parallelitätslimit begrenzt aktive Zustellungen und erhöht keine Worker-Kapazität. Läufe erzeugen dauerhafte Befehls- und Recovery-Nachweise. Veröffentlichung erstellt eine gesteuerte Datasource-Materialisierung; Promotion wählt eine unveränderliche Revision für Staging oder Produktion aus. Abgleichentscheidungen werden mit optimistischer Nebenläufigkeit als neue Revision gespeichert und verändern die geprüfte Fachzeile nicht. Löschen verhindert künftige Nutzung, während Lauf-, Bereitstellungs-, Herkunfts-, Audit- und Recovery-Nachweise ihrer Aufbewahrung folgen."
|
||||
),
|
||||
},
|
||||
"dataflow.execution-and-recovery": {
|
||||
"title": "Dataflow ausführen, veröffentlichen und wiederherstellen",
|
||||
"summary": "Gebundene Läufe, Umgebungsfreigaben, Ausgabeveröffentlichung, Abbruch, Abgleich und Nachweise betreiben.",
|
||||
"body": (
|
||||
"Jeder Lauf ist an eine unveränderliche Revision und einen Idempotenzschlüssel gebunden. Die Warteschlange erfasst handelnde Person, Befugnis, Umgebung, Fortschritt, Abbruch, Ausgabe und Recovery-Zustand. Reine Datenbankläufe werden atomar gespeichert. Die Veröffentlichung in eine gesteuerte Datasource nutzt Vorwärts-Recovery: Ein unbekanntes Anbieterergebnis wird vor einer Wiederholung abgeglichen, damit keine Ausgabe doppelt entsteht. Staging- und Produktionsfreigaben sind ausdrücklich und schreiben keine Revision um. Das Einfrieren einer Veröffentlichung versieht exakt die unveränderliche Ausgabe mit einer dauerhaften Bezeichnung; Daten werden weder kopiert noch von Aufbewahrungs-, Hold- und Zugriffsregeln der Datasource getrennt. Artefaktbasierte und Inline-Ausgaben liefern dieselben stabilen Veröffentlichungs-, Datasource- und Materialisierungsreferenzen. Warnungen und prüfpflichtige Zustände bleiben für Workflow sichtbar. Ein Abbruch ist nach Beginn externer Arbeit nur bestmöglich; der Abschlussnachweis unterscheidet gestoppt, abgeschlossen, fehlgeschlagen und abgleichpflichtig. Vor Annahme eines geplanten, ereignisbasierten oder eingereihten Laufs wird die Modulberechtigung des Mandanten geprüft. Eine Deaktivierung stoppt neue Annahmen und überlässt bereits angenommene Läufe einer ausdrücklichen Betriebsentscheidung. Reporting darf nur einen erfolgreichen veröffentlichten Lauf binden; Dataflow prüft Befugnis und Datasource-Zugriff erneut und liest exakt die protokollierte Materialisierung ohne erneute Parametrisierung oder Ausführung."
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def localize_documentation_topics(
|
||||
topics: Iterable[DocumentationTopic],
|
||||
) -> tuple[DocumentationTopic, ...]:
|
||||
return _localize_topics(topics, locale="de", translations=_TRANSLATIONS)
|
||||
@@ -0,0 +1,95 @@
|
||||
"""German translations for public structured documentation metadata."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
GERMAN_STRUCTURED_TRANSLATIONS: dict[str, dict[str, Any]] = {'dataflow.execution-and-recovery': {'consequence_classes': {'cancel_run': 'Stornierung von '
|
||||
'Anfragen; bereits '
|
||||
'anerkannte externe '
|
||||
'Effekte können '
|
||||
'verbleiben.',
|
||||
'promote_revision': 'Macht eine '
|
||||
'unveränderliche '
|
||||
'Revision in '
|
||||
'einer höheren '
|
||||
'Ausführungsumgebung '
|
||||
'geeignet.',
|
||||
'publish_output': 'Erstellt oder '
|
||||
'aktualisiert eine '
|
||||
'geregelte '
|
||||
'Datenquelle und '
|
||||
'fügt eine '
|
||||
'Materialisierung '
|
||||
'hinzu.',
|
||||
'queue_run': 'Erstellt einen '
|
||||
'dauerhaften asynchronen '
|
||||
'Befehls- und '
|
||||
'Autorisierungsnachweis.'}},
|
||||
'dataflow.reference.fields-and-consequences': {'consequence_classes': {'configure_trigger': 'Erstellt '
|
||||
'oder '
|
||||
'ändert '
|
||||
'einen '
|
||||
'Automatisierungsbefehl '
|
||||
'mit '
|
||||
'Revisions- '
|
||||
'und '
|
||||
'Autorisierungsnachweisen.',
|
||||
'delete_pipeline': 'Verhindert '
|
||||
'die '
|
||||
'zukünftige '
|
||||
'Verwendung, '
|
||||
'während '
|
||||
'beibehaltene '
|
||||
'Nachweise '
|
||||
'geregelt '
|
||||
'bleiben.',
|
||||
'derive_copy': 'Erstellt '
|
||||
'eine '
|
||||
'separat '
|
||||
'verwaltete '
|
||||
'Kopie, die '
|
||||
'an die '
|
||||
'Quellrevision '
|
||||
'und den '
|
||||
'Hash '
|
||||
'gebunden '
|
||||
'ist.',
|
||||
'rebase_copy': 'Hängt die '
|
||||
'genaue '
|
||||
'überprüfte '
|
||||
'Quellrevision '
|
||||
'an eine '
|
||||
'Scope-Kopie '
|
||||
'an, '
|
||||
'zeichnet '
|
||||
'die '
|
||||
'Herkunft '
|
||||
'des '
|
||||
'Reviewers '
|
||||
'auf und '
|
||||
'gibt sie '
|
||||
'in den '
|
||||
'Entwurf '
|
||||
'zurück.',
|
||||
'record_decision': 'Fügt '
|
||||
'eine '
|
||||
'vom '
|
||||
'handelnde '
|
||||
'Person '
|
||||
'zugewiesene '
|
||||
'unveränderliche '
|
||||
'Entscheidungsrevision '
|
||||
'gegen '
|
||||
'einen '
|
||||
'genauen '
|
||||
'Eingabe-Hash '
|
||||
'an.',
|
||||
'save_revision': 'Fügt '
|
||||
'eine '
|
||||
'unveränderliche '
|
||||
'Überarbeitung '
|
||||
'der '
|
||||
'Pipelinedefinition '
|
||||
'an.'}}}
|
||||
@@ -121,6 +121,7 @@ def definition_governance_payload(
|
||||
*,
|
||||
principal: ApiPrincipal,
|
||||
registry: object | None,
|
||||
source_update: Mapping[str, object] | None = None,
|
||||
) -> dict[str, object]:
|
||||
actions = {
|
||||
action: definition_decision(
|
||||
@@ -142,6 +143,25 @@ def definition_governance_payload(
|
||||
"derived_from_pipeline_id": pipeline.derived_from_pipeline_id,
|
||||
"derived_from_revision": pipeline.derived_from_revision,
|
||||
"derived_from_hash": pipeline.derived_from_hash,
|
||||
"source_available": bool(
|
||||
source_update and source_update.get("source_available") is True
|
||||
),
|
||||
"source_name": (
|
||||
source_update.get("source_name") if source_update else None
|
||||
),
|
||||
"source_current_revision": (
|
||||
source_update.get("source_current_revision")
|
||||
if source_update
|
||||
else None
|
||||
),
|
||||
"source_current_hash": (
|
||||
source_update.get("source_current_hash")
|
||||
if source_update
|
||||
else None
|
||||
),
|
||||
"update_available": bool(
|
||||
source_update and source_update.get("update_available") is True
|
||||
),
|
||||
"derivation_provenance": dict(pipeline.derivation_provenance),
|
||||
"actions": actions,
|
||||
}
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from govoplan_core.core.modules import with_documentation_structured_translations
|
||||
from govoplan_dataflow.backend.german_structured_documentation import GERMAN_STRUCTURED_TRANSLATIONS
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from govoplan_core.core.module_guards import (
|
||||
@@ -17,6 +20,8 @@ from govoplan_core.core.dataflows import (
|
||||
CAPABILITY_DATAFLOW_TRIGGER_DISPATCHER,
|
||||
)
|
||||
from govoplan_core.core.modules import (
|
||||
CapabilityDocumentation,
|
||||
DocumentationCondition,
|
||||
DocumentationTopic,
|
||||
FrontendModule,
|
||||
FrontendRoute,
|
||||
@@ -27,6 +32,7 @@ from govoplan_core.core.modules import (
|
||||
ModuleManifest,
|
||||
NavItem,
|
||||
PermissionDefinition,
|
||||
ProductAreaContribution,
|
||||
RoleTemplate,
|
||||
)
|
||||
from govoplan_core.core.provider_governance import declared_module_architecture
|
||||
@@ -46,11 +52,18 @@ from govoplan_core.core.search import SearchSourceProviderRegistration
|
||||
from govoplan_core.core.views import ViewSurface
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_dataflow.backend.db import models as dataflow_models
|
||||
from govoplan_dataflow.backend.dsar_provider import (
|
||||
DATAFLOW_DSAR_CAPABILITY,
|
||||
DataflowDsarProvider,
|
||||
)
|
||||
from govoplan_dataflow.backend.german_documentation import (
|
||||
localize_documentation_topics,
|
||||
)
|
||||
|
||||
|
||||
MODULE_ID = "dataflow"
|
||||
MODULE_NAME = "Dataflow"
|
||||
MODULE_VERSION = "0.1.17"
|
||||
MODULE_VERSION = "0.1.25"
|
||||
|
||||
READ_SCOPE = "dataflow:pipeline:read"
|
||||
WRITE_SCOPE = "dataflow:pipeline:write"
|
||||
@@ -140,7 +153,140 @@ ROLE_TEMPLATES = (
|
||||
),
|
||||
)
|
||||
|
||||
DOCUMENTATION = (
|
||||
DOCUMENTATION = localize_documentation_topics((
|
||||
DocumentationTopic(
|
||||
id="dataflow.csv-source-fidelity",
|
||||
title="Import CSV without silently changing its values",
|
||||
summary="Choose text preservation or explicit legacy inference before creating a durable datasource.",
|
||||
body=(
|
||||
"The CSV import dialog defaults to Preserve text (no automatic conversion). Field whitespace, decimal digits, large identifiers, boolean-looking text and explicit empty records remain strings. "
|
||||
"Choose Infer types (legacy) only when you want the existing numeric/boolean conversion and empty-row rules. JSON imports are unchanged. Existing API clients omitting csv_value_mode retain legacy_typed behavior. "
|
||||
"The imported datasource is a deliberate durable upload, not a retained preview. Datasources owns its original UTF-8 CSV text, parsed rows, approval evidence, immutable materialization and retention; Core verifies that original input and parsed scalar types/values agree. "
|
||||
"Both original and parsed content are bounded to 5 MB and the table to 10,000 rows. Original-source export is available through the Datasources administrator API only with unrestricted current and historical visibility, never through catalogue metadata. "
|
||||
"Missing older originals cannot be reconstructed. Exact text mode can change the inferred schema to strings; review downstream numeric comparisons and conversions before adopting the new source."
|
||||
),
|
||||
layer="always", documentation_types=("user", "admin"), audience=("user", "module_admin", "operator"), order=8,
|
||||
translations={"de": {
|
||||
"title": "CSV importieren, ohne Werte unbemerkt zu ändern",
|
||||
"summary": "Vor dem dauerhaften Import zwischen Texterhalt und ausdrücklicher bisheriger Typableitung wählen.",
|
||||
"body": (
|
||||
"Der CSV-Import wählt standardmäßig Text erhalten (keine automatische Umwandlung). Leerzeichen in Werten, Dezimalstellen, große Kennungen, boolesch wirkender Text und ausdrücklich leere Datensätze bleiben Zeichenketten. "
|
||||
"Wählen Sie Typen ableiten (bisheriges Verhalten), wenn Sie die bisherige Zahlen-/Wahrheitswertumwandlung und Behandlung leerer Zeilen benötigen. JSON-Importe bleiben unverändert; API-Aufrufe ohne csv_value_mode behalten legacy_typed. "
|
||||
"Der Import ist eine bewusst dauerhafte Datenquelle und keine gespeicherte Vorschau. Datasources verantwortet Originaltext als UTF-8, verarbeitete Zeilen, Freigabenachweis, unveränderliche Materialisierung und Aufbewahrung. Core prüft die Übereinstimmung von Quelltext sowie genauen Typen und Werten. "
|
||||
"Original und verarbeiteter Inhalt sind jeweils auf 5 MB, die Tabelle auf 10.000 Zeilen begrenzt. Das Original kann nur über die Datasources-Administrations-API mit uneingeschränkter aktueller und historischer Sichtbarkeit abgerufen werden, nicht über Katalogmetadaten. "
|
||||
"Fehlende ältere Originale lassen sich nicht rekonstruieren. Der Textmodus kann Spalten zu Zeichenketten machen; prüfen Sie deshalb nachgelagerte Zahlenvergleiche und Umwandlungen vor der Übernahme."
|
||||
),
|
||||
}},
|
||||
),
|
||||
DocumentationTopic(
|
||||
id="dataflow.save-completion",
|
||||
title="Editing while a pipeline save completes",
|
||||
summary="Keep newer local edits separate from the immutable revision accepted by the server.",
|
||||
body=(
|
||||
"You may continue editing a pipeline while Save is pending. The accepted server revision becomes the saved baseline; "
|
||||
"fields changed since submission remain in the local unsaved draft and require another explicit save. Graphs are kept as whole values, "
|
||||
"not merged or reordered node by node. Save-and-leave does not navigate while newer edits remain unsaved. "
|
||||
"A second save uses the revision actually accepted by the first request; duplicate concurrent save submissions are blocked. "
|
||||
"Selecting, replacing or discarding a draft, leaving the page, or changing authentication context prevents an old completion from replacing the current editor. "
|
||||
"Such a request may already have succeeded on the server: reload and review before retrying if the context changed. "
|
||||
"Ordinary session refreshes with the same identity, credentials and permissions keep accepted IDs and revisions; cosmetic profile changes do not interrupt saving. "
|
||||
"If a save was accepted across a real authorization change, further saves in that edit session are blocked until the draft is replaced after review, so a new pipeline is not created twice. "
|
||||
"Revision conflicts retain the local draft and require review; neither the UI nor administrators automatically overwrite a conflicting server revision. "
|
||||
"Current permissions and governance remain server-enforced. No preview rows are persisted by this editor behavior."
|
||||
),
|
||||
layer="always", documentation_types=("user", "admin"), audience=("user", "module_admin", "operator"), order=7,
|
||||
translations={"de": {
|
||||
"title": "Während des Speicherns einer Pipeline weiterarbeiten",
|
||||
"summary": "Neuere lokale Änderungen von der unveränderlichen, serverseitig angenommenen Revision trennen.",
|
||||
"body": (
|
||||
"Während Speichern läuft, können Sie die Pipeline weiter bearbeiten. Die angenommene Serverrevision wird zum gespeicherten Vergleichsstand; "
|
||||
"seit dem Absenden geänderte Felder bleiben im lokalen, ungespeicherten Entwurf und benötigen einen weiteren ausdrücklichen Speichervorgang. "
|
||||
"Graphen bleiben vollständige Werte und werden nicht knotenweise zusammengeführt oder umsortiert. Speichern und Verlassen navigiert nicht, solange neuere Änderungen ungespeichert sind. "
|
||||
"Ein zweiter Speichervorgang verwendet die tatsächlich angenommene Revision des ersten; doppelte gleichzeitige Speicheranfragen werden blockiert. "
|
||||
"Auswahl, Ersetzen oder Verwerfen eines Entwurfs, Verlassen der Seite oder ein geänderter Authentifizierungskontext verhindern, dass ein altes Ergebnis den aktuellen Editor ersetzt. "
|
||||
"Die Anfrage kann auf dem Server bereits erfolgreich gewesen sein: Nach einem Kontextwechsel vor einem erneuten Versuch neu laden und prüfen. "
|
||||
"Gewöhnliche Sitzungsaktualisierungen mit gleicher Identität, gleichen Zugangsdaten und Rechten behalten angenommene IDs und Revisionen; rein optische Profiländerungen unterbrechen das Speichern nicht. "
|
||||
"Wurde ein Speichervorgang während einer tatsächlichen Berechtigungsänderung angenommen, bleiben weitere Speicheranfragen dieser Bearbeitungssitzung bis zum geprüften Ersetzen des Entwurfs gesperrt, damit keine Pipeline doppelt entsteht. "
|
||||
"Bei Revisionskonflikten bleibt der lokale Entwurf erhalten und muss geprüft werden; weder Oberfläche noch Administratoren überschreiben automatisch eine widersprechende Serverrevision. "
|
||||
"Aktuelle Rechte und Governance werden weiterhin serverseitig geprüft. Dieses Editorverhalten speichert keine Vorschauzeilen dauerhaft."
|
||||
),
|
||||
}},
|
||||
),
|
||||
DocumentationTopic(
|
||||
id="dataflow.reference-worker-limits",
|
||||
title="Reference execution process limits",
|
||||
summary="Contain expensive expressions and intermediate allocations without changing datasource authority.",
|
||||
body=(
|
||||
"Reference previews and reference development runs evaluate in a fresh disposable process, including regex and aggregate intermediate allocations. "
|
||||
"The existing row, node and per-result byte checks remain. The process additionally enforces the request's wall-clock and memory budgets "
|
||||
"(default preview: 2 seconds and 256 MiB virtual address space), rounded-up CPU seconds, no regular-file output, and 32 MiB per data-only input/result transport. "
|
||||
"Supported process budgets are at most 600 seconds and 8 GiB; unsupported controls or exceeded limits fail with structured backend.process diagnostics, never inline fallback. "
|
||||
"Datasource authorization and bounded source reads remain in the parent, including nested subflow sources; sessions and credentials are not passed to the child. "
|
||||
"Reference source collection also checks a cumulative 32 MiB typed-data budget before constructing another columnar copy; providers retain their separate per-read limits. "
|
||||
"Completed node diagnostics survive ordinary evaluation errors; a killed worker returns no partial rows or invented node progress. "
|
||||
"GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY limits shared isolated-work admission per API/worker process, default 1; busy capacity is retryable. "
|
||||
"It is not a fleet-wide quota or arbitrary-code sandbox. Cancellation checks before/after reference runs remain; hard wall limits stop an unresponsive expression. "
|
||||
"Staging/production still require the separate DuckDB backend and are not converted to reference execution."
|
||||
),
|
||||
layer="static", documentation_types=("user", "admin"), audience=("user", "module_admin", "operator"), order=6,
|
||||
translations={"de": {
|
||||
"title": "Prozessgrenzen der Referenzausführung",
|
||||
"summary": "Aufwendige Ausdrücke und Zwischenspeicher begrenzen, ohne Datenquellenrechte zu verändern.",
|
||||
"body": (
|
||||
"Referenz-Vorschauen und Referenz-Entwicklungsläufe werten Ausdrücke in einem frischen, kurzlebigen Prozess aus, einschließlich regulärer Ausdrücke und großer Zwischenergebnisse. "
|
||||
"Bestehende Zeilen-, Knoten- und Ergebnis-Bytegrenzen bleiben bestehen. Zusätzlich gelten das Laufzeit- und Speicherbudget der Anfrage "
|
||||
"(Vorschau standardmäßig 2 Sekunden und 256 MiB virtueller Adressraum), aufgerundete CPU-Sekunden, keine regulären Ausgabedateien und je 32 MiB für den reinen Datentransport. "
|
||||
"Prozessbudgets unterstützen höchstens 600 Sekunden und 8 GiB. Fehlende Betriebssystemkontrollen oder überschrittene Grenzen erzeugen strukturierte backend.process-Diagnosen, ohne Ausweichbetrieb im Hauptprozess. "
|
||||
"Datenquellenrechte und begrenzte Quellabrufe werden im Hauptprozess geprüft, auch für verschachtelte Teilflüsse; Sitzungen und Zugangsdaten gelangen nicht in den Kindprozess. "
|
||||
"Referenz-Quellabrufe prüfen außerdem zusammen höchstens 32 MiB typisierte Daten, bevor eine weitere spaltenweise Kopie entsteht; getrennte Abrufgrenzen der Anbieter bleiben bestehen. "
|
||||
"Gewöhnliche Auswertungsfehler behalten bereits abgeschlossene Knotendiagnosen. Ein gestoppter Prozess liefert keine Teilzeilen und keinen erfundenen Knotenfortschritt. "
|
||||
"GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY begrenzt gemeinsam genutzte isolierte Arbeit je API-/Worker-Prozess, standardmäßig 1; bei Auslastung ist ein erneuter Versuch möglich. "
|
||||
"Dies ist weder eine systemweite Quote noch eine Sandbox für beliebigen Code. Abbruchprüfungen vor und nach Referenzläufen bleiben erhalten; harte Laufzeitgrenzen stoppen hängende Ausdrücke. "
|
||||
"Staging und Produktion benötigen weiterhin das gesonderte DuckDB-Backend und wechseln nicht zur Referenzausführung."
|
||||
),
|
||||
}},
|
||||
),
|
||||
DocumentationTopic(
|
||||
id="dataflow.workspace-layout",
|
||||
title="Dataflow workspace actions",
|
||||
summary="Find collection-wide commands in their consistent workspace position.",
|
||||
body="The workspace documentation book sits beside Pipelines; automation and run help sits beside "
|
||||
"the corresponding dialog title, and field help stays with its label. "
|
||||
"Reload and New pipeline use the persistent full-width workspace header at the upper right; Reload sits immediately before creation. Selecting a record, changing filters, or opening an editor does not move these collection-wide commands into the left pane. SQL editing, validation, previews, saving, and execution keep their existing editor scope and bounded safety rules. Existing permissions, disabled-state rules, and unsaved-change guards still apply. Administrators configure authority through the existing permission system; no new permission or automatic operation is introduced.",
|
||||
layer="static",
|
||||
documentation_types=("user", "admin"),
|
||||
audience=("user", "module_admin", "operator"),
|
||||
order=5,
|
||||
translations={"de": {
|
||||
"title": "Datenflüsse: Aktionen im Arbeitsbereich",
|
||||
"summary": "Sammlungsweite Aktionen an ihrer einheitlichen Position im Arbeitsbereich finden.",
|
||||
"body": "Das Dokumentationsbuch des Arbeitsbereichs steht neben Pipelines; Hilfe zu Automatisierung "
|
||||
"und Ausführung steht neben dem jeweiligen Dialogtitel, und Feldhilfe bleibt bei der "
|
||||
"Feldbezeichnung. "
|
||||
"Neu laden und Neue Pipeline stehen oben rechts in der dauerhaft sichtbaren, arbeitsbereichsweiten Leiste; Neu laden steht unmittelbar vor dem Anlegen. Auswahl, Filterwechsel und Bearbeitung verschieben diese sammlungsweiten Aktionen nicht in den linken Bereich. SQL-Bearbeitung, Validierung, Vorschau, Speichern und Ausführung behalten ihren bisherigen Editorbereich und ihre begrenzenden Sicherheitsregeln. Bestehende Berechtigungen, Deaktivierungsregeln und der Schutz ungespeicherter Änderungen gelten weiterhin. Administratoren konfigurieren Rechte im bestehenden Berechtigungssystem; es entstehen weder neue Rechte noch automatische Vorgänge.",
|
||||
}},
|
||||
),
|
||||
DocumentationTopic(
|
||||
id="dataflow.data-subject-requests",
|
||||
title="Dataflow data-subject requests",
|
||||
summary="Minimize retained transformation detail without treating derived flows as authoritative subject records.",
|
||||
body=(
|
||||
"Dataflow matches exact tenant-scoped pipeline, revision, reconciliation, run, deployment, trigger, and delivery identifiers plus minimized account, identity, and membership attribution. Results never copy graphs, SQL, request or event payloads, reconciliation corrections, authorization snapshots, provenance bodies, errors, source details, hashes, credentials, or output rows. Dataflow does not scan arbitrary transformation content for a person; the authoritative input module must locate and correct subject facts. "
|
||||
"Explicitly identified terminal run and delivery detail can be minimized idempotently, and automation authority linked to the subject can be disabled and revoked. Definitions, reconciliation evidence, active work, deployments, broad pipeline packages, published Datasource outputs, and institutional attribution require authorized review or retention. Correct sources and refresh Datasource, Search, and Reporting derivatives after review."
|
||||
),
|
||||
layer="configured",
|
||||
documentation_types=("admin", "user"),
|
||||
audience=("user", "operator", "module_admin", "data_steward", "auditor"),
|
||||
order=74,
|
||||
related_modules=("core", "datasources", "reporting", "workflow_engine"),
|
||||
metadata={
|
||||
"help_contexts": [
|
||||
"dataflow.data-subject-requests",
|
||||
"dataflow.runs",
|
||||
"dataflow.triggers",
|
||||
],
|
||||
},
|
||||
),
|
||||
DocumentationTopic(
|
||||
id="dataflow.module-boundary",
|
||||
title="Dataflow module boundary",
|
||||
@@ -152,7 +298,12 @@ DOCUMENTATION = (
|
||||
"and credentials; Reporting owns analytical presentation and exports; "
|
||||
"Workflow owns orchestration and human handoffs; Risk Compliance owns sanctions review "
|
||||
"semantics and policy gates. User SQL is compiled into approved transforms and is never "
|
||||
"passed unchecked to a backing database."
|
||||
"passed unchecked to a backing database. Opening or reloading Dataflow does not "
|
||||
"start a pipeline. If the editor cannot be loaded after a development update, "
|
||||
"preserve unsaved work before reloading the browser. Administrators should "
|
||||
"distinguish frontend asset failures from pipeline API or access errors. "
|
||||
"Development and browser-conformance servers use separate dependency caches; "
|
||||
"an older server may need restarting after this configuration update."
|
||||
),
|
||||
layer="available",
|
||||
documentation_types=("admin", "user"),
|
||||
@@ -170,6 +321,19 @@ DOCUMENTATION = (
|
||||
"audit",
|
||||
),
|
||||
metadata={
|
||||
"kind": "workflow",
|
||||
"route": "/dataflow",
|
||||
"screen": "Dataflow",
|
||||
"prerequisites": [
|
||||
"You may read Dataflow pipeline definitions in the active tenant.",
|
||||
],
|
||||
"steps": [
|
||||
"Open Dataflow and select a pipeline or create an authorized draft.",
|
||||
"Inspect the graph, immutable revision, diagnostics, and declared source references.",
|
||||
"Use preview or run actions only when the effective permissions and Policy allow them.",
|
||||
],
|
||||
"outcome": "The pipeline remains a governed transformation definition with explicit module boundaries and source authority.",
|
||||
"verification": "Confirm the active revision, validation diagnostics, source references, and permitted actions in the Dataflow workspace.",
|
||||
"first_slice": (
|
||||
"Inline and governed datasources, union, join, filter, deduplication, select, "
|
||||
"typed expressions, conversion, quality and reconciliation, reusable subflows, "
|
||||
@@ -191,6 +355,27 @@ DOCUMENTATION = (
|
||||
"dataflow.state.read-only",
|
||||
],
|
||||
},
|
||||
conditions=(
|
||||
DocumentationCondition(
|
||||
required_modules=("dataflow",),
|
||||
required_scopes=(READ_SCOPE,),
|
||||
),
|
||||
),
|
||||
structured_translation_version="1",
|
||||
structured_translations={
|
||||
"de": {
|
||||
"prerequisites": [
|
||||
"Sie dürfen Dataflow-Pipeline-Definitionen im aktiven Mandanten lesen.",
|
||||
],
|
||||
"steps": [
|
||||
"Öffnen Sie Dataflow und wählen Sie eine Pipeline oder legen Sie einen autorisierten Entwurf an.",
|
||||
"Prüfen Sie Graph, unveränderliche Revision, Diagnosen und ausgewiesene Quellreferenzen.",
|
||||
"Verwenden Sie Vorschau- oder Laufaktionen nur, wenn wirksame Berechtigungen und Policy sie erlauben.",
|
||||
],
|
||||
"outcome": "Die Pipeline bleibt eine gesteuerte Transformationsdefinition mit ausdrücklichen Modulgrenzen und Quellenautorität.",
|
||||
"verification": "Prüfen Sie aktive Revision, Validierungsdiagnosen, Quellreferenzen und erlaubte Aktionen im Dataflow-Arbeitsbereich.",
|
||||
}
|
||||
},
|
||||
),
|
||||
DocumentationTopic(
|
||||
id="dataflow.reference.nodes-and-expressions",
|
||||
@@ -200,11 +385,26 @@ DOCUMENTATION = (
|
||||
"Every graph node declares typed inputs, configuration, output schema, and validation rules. "
|
||||
"Source nodes pin inline content or governed Datasource references; combine, filter, transform, "
|
||||
"quality, reconciliation, reusable-subflow, and output nodes remain explicit in the canonical graph. "
|
||||
"Reconciliation rows expose stable key hashes, explicit before/after values, and input hashes. A separate decision-table input can annotate exact matches, invalidate changed inputs, and report orphaned decisions without silently rewriting business data. "
|
||||
"Reusable subflows select a Policy-authorized immutable flow or template revision. The server resolves "
|
||||
"and pins its graph, source hash, Policy decision, and closed typed input/output contracts; caller-supplied "
|
||||
"graph snapshots are ignored, incompatible inputs fail validation, and nested reference cycles are rejected. "
|
||||
"Reconciliation rows expose stable key hashes, explicit before/after values, and input hashes. The "
|
||||
"review dialog records accept, reject, correct, or defer decisions in tenant-owned immutable decision "
|
||||
"sets. Their current projection is a fingerprinted Dataflow source; every superseded revision retains "
|
||||
"the actor, reason, exact input hash, time, and optional correction. A reconcile.decisions node can "
|
||||
"annotate exact matches, invalidate changed inputs, and report orphaned decisions without silently "
|
||||
"rewriting business data. "
|
||||
"Expressions use the typed Dataflow expression language and never execute arbitrary host or database "
|
||||
"code. Selecting a node may request a bounded intermediate preview; preview rows are transient, "
|
||||
"privacy-filtered for the actor, and are not retained as run output. SQL editing compiles into the same "
|
||||
"canonical graph, so unsupported statements are diagnostics rather than pass-through SQL. "
|
||||
"Reference previews retain the existing 1,000,000-byte serialized result limit per node. "
|
||||
"LPAD and RPAD reject target lengths above 1,000,000 characters before allocating padding, "
|
||||
"including oversized intermediate values inside LENGTH or SUBSTRING even if the final scalar would be small. "
|
||||
"Reduce the requested padding length; this fails the expression at its node instead of truncating data. "
|
||||
"Ordinary Unicode padding, null inputs, and in-budget truncation keep their existing behavior; "
|
||||
"the final byte check still accounts for JSON and multibyte character overhead. "
|
||||
"This allocation guard does not replace runtime or byte limits for other expression operations."
|
||||
),
|
||||
layer="available",
|
||||
documentation_types=("admin", "user"),
|
||||
@@ -212,12 +412,17 @@ DOCUMENTATION = (
|
||||
order=76,
|
||||
related_modules=("datasources", "connectors", "policy", "audit"),
|
||||
metadata={
|
||||
"kind": "reference",
|
||||
"help_contexts": [
|
||||
"dataflow.field.node-name",
|
||||
"dataflow.field.source",
|
||||
"dataflow.field.expression",
|
||||
"dataflow.field.schema",
|
||||
"dataflow.field.reusable-input-binding",
|
||||
"dataflow.field.subflow-reference",
|
||||
"dataflow.field.subflow-revision",
|
||||
"dataflow.action.preview-node",
|
||||
"dataflow.action.review-decisions",
|
||||
],
|
||||
},
|
||||
),
|
||||
@@ -228,10 +433,20 @@ DOCUMENTATION = (
|
||||
body=(
|
||||
"Scope determines ownership and Policy inheritance. Templates can be derived but not run; complete "
|
||||
"flows may be previewed, revisioned, automated, and executed when effective Policy allows it. Saving "
|
||||
"appends an immutable revision. A scoped copy pins its source revision and content hash. Triggers pin "
|
||||
"the revision and authorization grant, then re-evaluate authority for every delivery. Runs create "
|
||||
"appends an immutable revision. A scoped copy pins its source revision and content hash. A newer "
|
||||
"source revision is reported without changing the copy. Adopting it requires an exact reviewed source "
|
||||
"hash and a reason, appends an immutable copy revision, records the Policy decision and reviewer, and "
|
||||
"returns the copy to draft so runs and automation cannot use the changed graph before activation. Triggers pin "
|
||||
"the revision and authorization grant, then re-evaluate authority for every delivery. Allow runs is "
|
||||
"the definition-level admission boundary and does not grant a caller permission. A one-time run uses "
|
||||
"the configured tenant-local date and time. The missed-run policy either coalesces elapsed interval "
|
||||
"occurrences into one latest delivery or skips them; it never silently replays every missed occurrence. "
|
||||
"The concurrency limit bounds active deliveries for that trigger and does not increase tenant worker "
|
||||
"capacity. Runs create "
|
||||
"durable command and recovery evidence. Publishing creates a governed Datasource materialization, and "
|
||||
"environment promotion changes which immutable revision is eligible for staging or production runs. "
|
||||
"Recording a reconciliation decision uses optimistic concurrency and appends an immutable revision; "
|
||||
"it never mutates the reviewed business row. "
|
||||
"Deletion prevents future use while retained run, deployment, lineage, audit, and recovery evidence "
|
||||
"continues under its retention policy."
|
||||
),
|
||||
@@ -239,20 +454,36 @@ DOCUMENTATION = (
|
||||
documentation_types=("admin", "user"),
|
||||
audience=("operator", "module_admin", "power_user", "product_owner"),
|
||||
order=77,
|
||||
related_modules=("datasources", "workflow_engine", "notifications", "policy", "audit"),
|
||||
related_modules=(
|
||||
"datasources",
|
||||
"workflow_engine",
|
||||
"notifications",
|
||||
"policy",
|
||||
"audit",
|
||||
),
|
||||
metadata={
|
||||
"kind": "reference",
|
||||
"help_contexts": [
|
||||
"dataflow.field.scope",
|
||||
"dataflow.field.definition-kind",
|
||||
"dataflow.field.allow-runs",
|
||||
"dataflow.field.trigger-run-at",
|
||||
"dataflow.field.trigger-missed-runs",
|
||||
"dataflow.field.trigger-concurrency",
|
||||
"dataflow.field.rebase-reason",
|
||||
"dataflow.action.save",
|
||||
"dataflow.action.derive",
|
||||
"dataflow.action.rebase",
|
||||
"dataflow.action.trigger",
|
||||
"dataflow.action.record-decision",
|
||||
"dataflow.action.delete",
|
||||
],
|
||||
"consequence_classes": {
|
||||
"save_revision": "Appends an immutable pipeline definition revision.",
|
||||
"derive_copy": "Creates a separately governed copy pinned to the source revision and hash.",
|
||||
"rebase_copy": "Appends the exact reviewed source revision to a scoped copy, records reviewer provenance, and returns it to draft.",
|
||||
"configure_trigger": "Creates or changes an automation command with revision and authorization evidence.",
|
||||
"record_decision": "Appends an actor-attributed immutable decision revision against an exact input hash.",
|
||||
"delete_pipeline": "Prevents future use while retained evidence remains governed.",
|
||||
},
|
||||
},
|
||||
@@ -266,7 +497,11 @@ DOCUMENTATION = (
|
||||
"environment, progress, cancellation, output, and recovery state. Database-only runs commit atomically. "
|
||||
"Publication to a governed Datasource uses forward recovery: an unknown provider outcome is reconciled "
|
||||
"before retry so output is not duplicated. Staging and production promotion is explicit and does not "
|
||||
"rewrite a revision. Cancellation is best effort once external work has started; the final evidence "
|
||||
"rewrite a revision. Freezing a published state assigns a durable label to the exact immutable output; "
|
||||
"it does not copy or detach the data from Datasources retention, hold, and access rules. Artifact-backed "
|
||||
"outputs return the same stable publication, datasource, and materialization references as inline "
|
||||
"outputs. Datasource warnings and review-required states remain visible to Workflow instead of being "
|
||||
"collapsed into success. Cancellation is best effort once external work has started; the final evidence "
|
||||
"states whether work stopped, completed, failed, or requires operator reconciliation. Scheduled, event, "
|
||||
"and queued execution is partitioned by tenant module entitlement before a run is claimed. Disabling "
|
||||
"Dataflow stops new admission and leaves accepted runs available for an explicit operator decision."
|
||||
@@ -282,6 +517,7 @@ DOCUMENTATION = (
|
||||
"dataflow.runs",
|
||||
"dataflow.action.queue-run",
|
||||
"dataflow.action.publish",
|
||||
"dataflow.field.freeze-publication",
|
||||
"dataflow.action.promote-staging",
|
||||
"dataflow.action.promote-production",
|
||||
"dataflow.state.recovery-attention",
|
||||
@@ -294,7 +530,7 @@ DOCUMENTATION = (
|
||||
},
|
||||
},
|
||||
),
|
||||
)
|
||||
))
|
||||
|
||||
|
||||
def _dataflow_router(context: ModuleContext):
|
||||
@@ -312,6 +548,11 @@ def _run_provider(context: ModuleContext):
|
||||
return SqlDataflowRunLifecycleProvider(registry=context.registry)
|
||||
|
||||
|
||||
def _dsar_provider(context: ModuleContext) -> DataflowDsarProvider:
|
||||
del context
|
||||
return DataflowDsarProvider()
|
||||
|
||||
|
||||
def _run_worker(context: ModuleContext):
|
||||
from govoplan_dataflow.backend.run_worker import SqlDataflowRunWorker
|
||||
|
||||
@@ -356,9 +597,20 @@ def _tenant_summary(session, tenant_id: str) -> dict[str, int]:
|
||||
),
|
||||
"dataflow_trigger_deliveries": (
|
||||
session.query(dataflow_models.DataflowTriggerDelivery)
|
||||
.filter(dataflow_models.DataflowTriggerDelivery.tenant_id == tenant_id)
|
||||
.count()
|
||||
),
|
||||
"dataflow_reconciliation_decision_sets": (
|
||||
session.query(dataflow_models.DataflowReconciliationDecisionSet)
|
||||
.filter(
|
||||
dataflow_models.DataflowTriggerDelivery.tenant_id
|
||||
== tenant_id
|
||||
dataflow_models.DataflowReconciliationDecisionSet.tenant_id == tenant_id
|
||||
)
|
||||
.count()
|
||||
),
|
||||
"dataflow_reconciliation_decisions": (
|
||||
session.query(dataflow_models.DataflowReconciliationDecision)
|
||||
.filter(
|
||||
dataflow_models.DataflowReconciliationDecision.tenant_id == tenant_id
|
||||
)
|
||||
.count()
|
||||
),
|
||||
@@ -393,15 +645,22 @@ manifest = ModuleManifest(
|
||||
CAPABILITY_POLICY_DEFINITION_GOVERNANCE,
|
||||
),
|
||||
provides_interfaces=(
|
||||
ModuleInterfaceProvider(name="dataflow.pipeline_catalog", version=MODULE_VERSION),
|
||||
ModuleInterfaceProvider(name="dataflow.pipeline_preview", version=MODULE_VERSION),
|
||||
ModuleInterfaceProvider(
|
||||
name="dataflow.pipeline_catalog", version=MODULE_VERSION
|
||||
),
|
||||
ModuleInterfaceProvider(
|
||||
name="dataflow.pipeline_preview", version=MODULE_VERSION
|
||||
),
|
||||
ModuleInterfaceProvider(name="dataflow.run_lifecycle", version=MODULE_VERSION),
|
||||
ModuleInterfaceProvider(name="dataflow.run_worker", version=MODULE_VERSION),
|
||||
ModuleInterfaceProvider(name=CAPABILITY_DATAFLOW_DATASET_OUTPUT, version=MODULE_VERSION),
|
||||
ModuleInterfaceProvider(
|
||||
name=CAPABILITY_DATAFLOW_DATASET_OUTPUT, version=MODULE_VERSION
|
||||
),
|
||||
ModuleInterfaceProvider(
|
||||
name="dataflow.trigger_dispatcher",
|
||||
version=MODULE_VERSION,
|
||||
),
|
||||
ModuleInterfaceProvider(name=DATAFLOW_DSAR_CAPABILITY, version="0.1.0"),
|
||||
),
|
||||
requires_interfaces=(
|
||||
ModuleInterfaceRequirement(
|
||||
@@ -484,6 +743,17 @@ manifest = ModuleManifest(
|
||||
order=72,
|
||||
),
|
||||
),
|
||||
product_areas=(
|
||||
ProductAreaContribution(
|
||||
id="data-assurance",
|
||||
module_id=MODULE_ID,
|
||||
label="i18n:govoplan-core.product_area.data_assurance",
|
||||
icon="database-zap",
|
||||
description="i18n:govoplan-core.product_area.data_assurance_description",
|
||||
surface_ids=("dataflow.nav.dataflow", "dataflow.route.dataflow"),
|
||||
order=60,
|
||||
),
|
||||
),
|
||||
view_surfaces=(
|
||||
ViewSurface(
|
||||
id="dataflow.page",
|
||||
@@ -532,6 +802,14 @@ manifest = ModuleManifest(
|
||||
parent_id="dataflow.page",
|
||||
order=50,
|
||||
),
|
||||
ViewSurface(
|
||||
id="dataflow.decisions",
|
||||
module_id=MODULE_ID,
|
||||
kind="action",
|
||||
label="Reconciliation decisions",
|
||||
parent_id="dataflow.results",
|
||||
order=55,
|
||||
),
|
||||
ViewSurface(
|
||||
id="dataflow.triggers",
|
||||
module_id=MODULE_ID,
|
||||
@@ -566,6 +844,16 @@ manifest = ModuleManifest(
|
||||
CAPABILITY_DATAFLOW_RUN_LIFECYCLE: _run_provider,
|
||||
CAPABILITY_DATAFLOW_RUN_WORKER: _run_worker,
|
||||
CAPABILITY_DATAFLOW_TRIGGER_DISPATCHER: _trigger_provider,
|
||||
DATAFLOW_DSAR_CAPABILITY: _dsar_provider,
|
||||
},
|
||||
capability_documentation={
|
||||
DATAFLOW_DSAR_CAPABILITY: CapabilityDocumentation(
|
||||
label="Dataflow data-subject request provider",
|
||||
summary="Finds and minimizes subject-linked transformation state and automation authority.",
|
||||
contract_version="0.1.0",
|
||||
documentation_types=("admin", "user"),
|
||||
audience=("privacy_officer", "data_steward", "user"),
|
||||
),
|
||||
},
|
||||
search_sources=(
|
||||
SearchSourceProviderRegistration(
|
||||
@@ -580,6 +868,8 @@ manifest = ModuleManifest(
|
||||
script_location=str(Path(__file__).with_name("migrations") / "versions"),
|
||||
retirement_supported=True,
|
||||
retirement_provider=drop_table_retirement_provider(
|
||||
dataflow_models.DataflowReconciliationDecision,
|
||||
dataflow_models.DataflowReconciliationDecisionSet,
|
||||
dataflow_models.DataflowTriggerDelivery,
|
||||
dataflow_models.DataflowTrigger,
|
||||
dataflow_models.DataflowRun,
|
||||
@@ -597,6 +887,8 @@ manifest = ModuleManifest(
|
||||
persistent_table_uninstall_guard(
|
||||
dataflow_models.DataflowPipeline,
|
||||
dataflow_models.DataflowPipelineRevision,
|
||||
dataflow_models.DataflowReconciliationDecisionSet,
|
||||
dataflow_models.DataflowReconciliationDecision,
|
||||
dataflow_models.DataflowPipelineDeployment,
|
||||
dataflow_models.DataflowRun,
|
||||
dataflow_models.DataflowTrigger,
|
||||
@@ -614,8 +906,19 @@ manifest = ModuleManifest(
|
||||
known_limits=(
|
||||
"Execution adapters do not yet cover every declared node family.",
|
||||
),
|
||||
owned_concepts=("dataflow definition", "dataflow revision", "dataflow run", "transformation graph"),
|
||||
non_owned_concepts=("datasource binding", "connector transport", "report presentation", "workflow task"),
|
||||
owned_concepts=(
|
||||
"dataflow definition",
|
||||
"dataflow revision",
|
||||
"dataflow run",
|
||||
"reconciliation decision set",
|
||||
"transformation graph",
|
||||
),
|
||||
non_owned_concepts=(
|
||||
"datasource binding",
|
||||
"connector transport",
|
||||
"report presentation",
|
||||
"workflow task",
|
||||
),
|
||||
recovery_docs=("README.md", "docs/DURABLE_RUN_RECOVERY.md"),
|
||||
security_docs=("README.md",),
|
||||
operations_docs=("README.md",),
|
||||
@@ -623,6 +926,11 @@ manifest = ModuleManifest(
|
||||
)
|
||||
|
||||
|
||||
manifest = with_documentation_structured_translations(
|
||||
manifest, locale="de", translations=GERMAN_STRUCTURED_TRANSLATIONS
|
||||
)
|
||||
|
||||
|
||||
def get_manifest() -> ModuleManifest:
|
||||
return manifest
|
||||
|
||||
|
||||
+118
@@ -0,0 +1,118 @@
|
||||
"""Add durable Dataflow reconciliation decision sets.
|
||||
|
||||
Revision ID: a3d7f1c5e9b2
|
||||
Revises: f6c2a9d4e7b1
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "a3d7f1c5e9b2"
|
||||
down_revision = "f6c2a9d4e7b1"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"dataflow_reconciliation_decision_sets",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("pipeline_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("name", sa.String(length=300), nullable=False),
|
||||
sa.Column("node_id", sa.String(length=100), nullable=True),
|
||||
sa.Column("resource_revision", sa.Integer(), nullable=False),
|
||||
sa.Column("created_by", sa.String(length=255), nullable=True),
|
||||
sa.Column("updated_by", sa.String(length=255), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["pipeline_id"],
|
||||
["dataflow_pipelines.id"],
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint(
|
||||
"tenant_id",
|
||||
"pipeline_id",
|
||||
"name",
|
||||
name="uq_dataflow_decision_sets_pipeline_name",
|
||||
),
|
||||
)
|
||||
op.create_index(
|
||||
"ix_dataflow_decision_sets_tenant_id",
|
||||
"dataflow_reconciliation_decision_sets",
|
||||
["tenant_id"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index(
|
||||
"ix_dataflow_decision_sets_pipeline_id",
|
||||
"dataflow_reconciliation_decision_sets",
|
||||
["pipeline_id"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index(
|
||||
"ix_dataflow_decision_sets_tenant_pipeline",
|
||||
"dataflow_reconciliation_decision_sets",
|
||||
["tenant_id", "pipeline_id"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_table(
|
||||
"dataflow_reconciliation_decisions",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("decision_set_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("revision", sa.Integer(), nullable=False),
|
||||
sa.Column("key_hash", sa.String(length=64), nullable=False),
|
||||
sa.Column("input_hash", sa.String(length=64), nullable=False),
|
||||
sa.Column("action", sa.String(length=20), nullable=False),
|
||||
sa.Column("reason", sa.Text(), nullable=False),
|
||||
sa.Column("correction", sa.JSON(), nullable=True),
|
||||
sa.Column("actor_ref", sa.String(length=255), nullable=False),
|
||||
sa.Column("decided_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["decision_set_id"],
|
||||
["dataflow_reconciliation_decision_sets.id"],
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint(
|
||||
"decision_set_id",
|
||||
"revision",
|
||||
name="uq_dataflow_reconciliation_decision_revision",
|
||||
),
|
||||
)
|
||||
op.create_index(
|
||||
"ix_dataflow_reconciliation_decisions_tenant_id",
|
||||
"dataflow_reconciliation_decisions",
|
||||
["tenant_id"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index(
|
||||
"ix_dataflow_reconciliation_decisions_decision_set_id",
|
||||
"dataflow_reconciliation_decisions",
|
||||
["decision_set_id"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index(
|
||||
"ix_dataflow_reconciliation_decisions_key_hash",
|
||||
"dataflow_reconciliation_decisions",
|
||||
["key_hash"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index(
|
||||
"ix_dataflow_reconciliation_decisions_current",
|
||||
"dataflow_reconciliation_decisions",
|
||||
["tenant_id", "decision_set_id", "key_hash", "revision"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("dataflow_reconciliation_decisions")
|
||||
op.drop_table("dataflow_reconciliation_decision_sets")
|
||||
@@ -613,14 +613,13 @@ _NODE_TYPES = (
|
||||
type="subflow",
|
||||
category="transform",
|
||||
label="Reusable subflow",
|
||||
description="Run a pinned parameterized template snapshot as one node.",
|
||||
description="Run a Policy-authorized, server-resolved immutable definition revision as one node.",
|
||||
icon="boxes",
|
||||
input_ports=(NodePortDefinition(id="input", label="Input"),),
|
||||
config_fields=(
|
||||
NodeConfigField(id="template_ref", label="Template reference", kind="text", required=True),
|
||||
NodeConfigField(id="template_version", label="Template version", kind="text", required=True),
|
||||
NodeConfigField(id="parameters", label="Parameters", kind="json", required=True),
|
||||
NodeConfigField(id="graph", label="Pinned graph", kind="json", required=True),
|
||||
),
|
||||
default_config={
|
||||
"template_ref": "",
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
"""Shared existing limits for reference-preview results and allocating expressions."""
|
||||
|
||||
MAX_RESULT_BYTES = 1_000_000
|
||||
@@ -0,0 +1,380 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from collections.abc import Sequence
|
||||
|
||||
from sqlalchemy import and_, func, select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.orm import Session, joinedload, selectinload
|
||||
|
||||
from govoplan_core.core.concurrency import claim_revision
|
||||
from govoplan_core.security.time import utc_now
|
||||
from govoplan_dataflow.backend.db.models import (
|
||||
DataflowPipeline,
|
||||
DataflowReconciliationDecision,
|
||||
DataflowReconciliationDecisionSet,
|
||||
)
|
||||
|
||||
|
||||
DECISION_SET_REF_PREFIX = "dataflow-decision-set:"
|
||||
DECISION_REF_PREFIX = "dataflow-decision:"
|
||||
DECISION_ACTIONS = frozenset({"accept", "reject", "correct", "defer"})
|
||||
|
||||
|
||||
class ReconciliationDecisionError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class ReconciliationDecisionNotFoundError(ReconciliationDecisionError):
|
||||
pass
|
||||
|
||||
|
||||
class ReconciliationDecisionConflictError(ReconciliationDecisionError):
|
||||
pass
|
||||
|
||||
|
||||
def list_decision_sets(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
pipeline_id: str | None = None,
|
||||
include_decisions: bool = True,
|
||||
) -> tuple[DataflowReconciliationDecisionSet, ...]:
|
||||
options = [joinedload(DataflowReconciliationDecisionSet.pipeline)]
|
||||
if include_decisions:
|
||||
options.append(selectinload(DataflowReconciliationDecisionSet.decisions))
|
||||
statement = (
|
||||
select(DataflowReconciliationDecisionSet)
|
||||
.options(*options)
|
||||
.where(DataflowReconciliationDecisionSet.tenant_id == tenant_id)
|
||||
.order_by(
|
||||
DataflowReconciliationDecisionSet.name,
|
||||
DataflowReconciliationDecisionSet.id,
|
||||
)
|
||||
)
|
||||
if pipeline_id:
|
||||
statement = statement.where(
|
||||
DataflowReconciliationDecisionSet.pipeline_id == pipeline_id
|
||||
)
|
||||
return tuple(session.scalars(statement).all())
|
||||
|
||||
|
||||
def create_decision_set(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
pipeline_id: str,
|
||||
name: str,
|
||||
node_id: str | None,
|
||||
actor_ref: str,
|
||||
) -> DataflowReconciliationDecisionSet:
|
||||
pipeline = session.get(DataflowPipeline, pipeline_id)
|
||||
if (
|
||||
pipeline is None
|
||||
or pipeline.tenant_id not in {None, tenant_id}
|
||||
or pipeline.deleted_at is not None
|
||||
):
|
||||
raise ReconciliationDecisionNotFoundError("Dataflow pipeline not found.")
|
||||
item = DataflowReconciliationDecisionSet(
|
||||
tenant_id=tenant_id,
|
||||
pipeline_id=pipeline_id,
|
||||
name=name.strip(),
|
||||
node_id=node_id,
|
||||
resource_revision=1,
|
||||
created_by=actor_ref,
|
||||
updated_by=actor_ref,
|
||||
)
|
||||
session.add(item)
|
||||
try:
|
||||
session.flush()
|
||||
except IntegrityError as exc:
|
||||
raise ReconciliationDecisionConflictError(
|
||||
"A decision set with this name already exists for the pipeline."
|
||||
) from exc
|
||||
return item
|
||||
|
||||
|
||||
def get_decision_set(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
decision_set_id: str,
|
||||
include_decisions: bool = True,
|
||||
) -> DataflowReconciliationDecisionSet:
|
||||
options = [joinedload(DataflowReconciliationDecisionSet.pipeline)]
|
||||
if include_decisions:
|
||||
options.append(selectinload(DataflowReconciliationDecisionSet.decisions))
|
||||
item = session.scalar(
|
||||
select(DataflowReconciliationDecisionSet)
|
||||
.options(*options)
|
||||
.where(
|
||||
DataflowReconciliationDecisionSet.id == decision_set_id,
|
||||
DataflowReconciliationDecisionSet.tenant_id == tenant_id,
|
||||
)
|
||||
)
|
||||
if item is None:
|
||||
raise ReconciliationDecisionNotFoundError(
|
||||
"Reconciliation decision set not found."
|
||||
)
|
||||
return item
|
||||
|
||||
|
||||
def record_decision(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
decision_set_id: str,
|
||||
expected_revision: int,
|
||||
key_hash: str,
|
||||
input_hash: str,
|
||||
action: str,
|
||||
reason: str,
|
||||
correction: dict[str, object] | None,
|
||||
actor_ref: str,
|
||||
) -> DataflowReconciliationDecisionSet:
|
||||
item = get_decision_set(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
decision_set_id=decision_set_id,
|
||||
)
|
||||
if action not in DECISION_ACTIONS:
|
||||
raise ReconciliationDecisionError("Unsupported reconciliation action.")
|
||||
next_revision = claim_revision(
|
||||
session,
|
||||
model=DataflowReconciliationDecisionSet,
|
||||
filters=(
|
||||
DataflowReconciliationDecisionSet.id == decision_set_id,
|
||||
DataflowReconciliationDecisionSet.tenant_id == tenant_id,
|
||||
),
|
||||
revision_attribute="resource_revision",
|
||||
expected_revision=expected_revision,
|
||||
resource_type="dataflow_reconciliation_decision_set",
|
||||
resource_id=decision_set_id,
|
||||
)
|
||||
item.resource_revision = next_revision
|
||||
item.updated_by = actor_ref
|
||||
item.decisions.append(
|
||||
DataflowReconciliationDecision(
|
||||
tenant_id=tenant_id,
|
||||
revision=next_revision,
|
||||
key_hash=key_hash,
|
||||
input_hash=input_hash,
|
||||
action=action,
|
||||
reason=reason.strip(),
|
||||
correction=dict(correction) if correction else None,
|
||||
actor_ref=actor_ref,
|
||||
decided_at=utc_now(),
|
||||
)
|
||||
)
|
||||
session.flush()
|
||||
return item
|
||||
|
||||
|
||||
def current_decisions(
|
||||
item: DataflowReconciliationDecisionSet,
|
||||
) -> tuple[DataflowReconciliationDecision, ...]:
|
||||
current: dict[str, DataflowReconciliationDecision] = {}
|
||||
for decision in sorted(item.decisions, key=lambda value: value.revision):
|
||||
current[decision.key_hash] = decision
|
||||
return tuple(
|
||||
sorted(
|
||||
current.values(),
|
||||
key=lambda value: (value.key_hash, value.revision),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def decision_rows(
|
||||
item: DataflowReconciliationDecisionSet,
|
||||
) -> tuple[dict[str, object], ...]:
|
||||
return tuple(_decision_row(value) for value in current_decisions(item))
|
||||
|
||||
|
||||
def decision_set_fingerprint(
|
||||
item: DataflowReconciliationDecisionSet,
|
||||
) -> str:
|
||||
payload = {
|
||||
"decision_set_id": item.id,
|
||||
"resource_revision": item.resource_revision,
|
||||
}
|
||||
return "sha256:" + hashlib.sha256(
|
||||
json.dumps(
|
||||
payload,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
ensure_ascii=True,
|
||||
).encode("utf-8")
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def decision_set_payload(
|
||||
item: DataflowReconciliationDecisionSet,
|
||||
*,
|
||||
include_decisions: bool = True,
|
||||
) -> dict[str, object]:
|
||||
current = current_decisions(item) if include_decisions else ()
|
||||
history = (
|
||||
sorted(item.decisions, key=lambda value: value.revision)
|
||||
if include_decisions
|
||||
else ()
|
||||
)
|
||||
return {
|
||||
"ref": f"{DECISION_SET_REF_PREFIX}{item.id}",
|
||||
"id": item.id,
|
||||
"pipeline_id": item.pipeline_id,
|
||||
"name": item.name,
|
||||
"node_id": item.node_id,
|
||||
"resource_revision": item.resource_revision,
|
||||
"etag": item.strong_etag,
|
||||
"fingerprint": decision_set_fingerprint(item),
|
||||
"decisions_included": include_decisions,
|
||||
"current_decisions": [_decision_payload(value) for value in current],
|
||||
"history": [
|
||||
_decision_payload(value)
|
||||
for value in history
|
||||
],
|
||||
"created_by": item.created_by,
|
||||
"updated_by": item.updated_by,
|
||||
"created_at": item.created_at,
|
||||
"updated_at": item.updated_at,
|
||||
}
|
||||
|
||||
|
||||
def decision_set_source_payload(
|
||||
item: DataflowReconciliationDecisionSet,
|
||||
) -> dict[str, object]:
|
||||
return {
|
||||
"ref": f"{DECISION_SET_REF_PREFIX}{item.id}",
|
||||
"provider": "dataflow.reconciliation_decisions",
|
||||
"source_name": _source_name(item.name, item.id),
|
||||
"name": item.name,
|
||||
"description": "Current immutable reconciliation decisions; prior revisions remain in decision history.",
|
||||
"mode": "static",
|
||||
"columns": [
|
||||
{"name": "key_hash", "data_type": "string", "nullable": False},
|
||||
{"name": "input_hash", "data_type": "string", "nullable": False},
|
||||
{"name": "decision_ref", "data_type": "string", "nullable": False},
|
||||
{"name": "action", "data_type": "string", "nullable": False},
|
||||
{"name": "actor_ref", "data_type": "string", "nullable": False},
|
||||
{"name": "decided_at", "data_type": "datetime", "nullable": False},
|
||||
{"name": "reason", "data_type": "string", "nullable": False},
|
||||
{"name": "correction", "data_type": "object", "nullable": True},
|
||||
],
|
||||
"schema_version": "1",
|
||||
"fingerprint": decision_set_fingerprint(item),
|
||||
"row_count": None,
|
||||
"byte_count": None,
|
||||
"updated_at": item.updated_at,
|
||||
"capabilities": ["preview", "read", "immutable_history"],
|
||||
}
|
||||
|
||||
|
||||
def current_decision_rows(
|
||||
session: Session,
|
||||
*,
|
||||
decision_set_id: str,
|
||||
limit: int,
|
||||
) -> tuple[tuple[dict[str, object], ...], int]:
|
||||
bounded_limit = max(1, limit)
|
||||
latest = (
|
||||
select(
|
||||
DataflowReconciliationDecision.key_hash.label("key_hash"),
|
||||
func.max(DataflowReconciliationDecision.revision).label("revision"),
|
||||
)
|
||||
.where(
|
||||
DataflowReconciliationDecision.decision_set_id == decision_set_id
|
||||
)
|
||||
.group_by(DataflowReconciliationDecision.key_hash)
|
||||
.subquery()
|
||||
)
|
||||
statement = (
|
||||
select(DataflowReconciliationDecision)
|
||||
.join(
|
||||
latest,
|
||||
and_(
|
||||
DataflowReconciliationDecision.key_hash == latest.c.key_hash,
|
||||
DataflowReconciliationDecision.revision == latest.c.revision,
|
||||
),
|
||||
)
|
||||
.where(
|
||||
DataflowReconciliationDecision.decision_set_id == decision_set_id
|
||||
)
|
||||
.order_by(DataflowReconciliationDecision.key_hash)
|
||||
.limit(bounded_limit)
|
||||
)
|
||||
records = tuple(session.scalars(statement))
|
||||
total = int(
|
||||
session.scalar(
|
||||
select(func.count(func.distinct(DataflowReconciliationDecision.key_hash))).where(
|
||||
DataflowReconciliationDecision.decision_set_id == decision_set_id
|
||||
)
|
||||
)
|
||||
or 0
|
||||
)
|
||||
return tuple(_decision_row(value) for value in records), total
|
||||
|
||||
|
||||
def decision_set_id_from_ref(value: str) -> str | None:
|
||||
cleaned = str(value or "").strip()
|
||||
if not cleaned.startswith(DECISION_SET_REF_PREFIX):
|
||||
return None
|
||||
identifier = cleaned[len(DECISION_SET_REF_PREFIX) :]
|
||||
return identifier or None
|
||||
|
||||
|
||||
def _decision_row(
|
||||
value: DataflowReconciliationDecision,
|
||||
) -> dict[str, object]:
|
||||
return {
|
||||
"key_hash": value.key_hash,
|
||||
"input_hash": value.input_hash,
|
||||
"decision_ref": f"{DECISION_REF_PREFIX}{value.id}",
|
||||
"action": value.action,
|
||||
"actor_ref": value.actor_ref,
|
||||
"decided_at": value.decided_at.isoformat(),
|
||||
"reason": value.reason,
|
||||
"correction": dict(value.correction) if value.correction else None,
|
||||
}
|
||||
|
||||
|
||||
def _decision_payload(
|
||||
value: DataflowReconciliationDecision,
|
||||
) -> dict[str, object]:
|
||||
return {
|
||||
"ref": f"{DECISION_REF_PREFIX}{value.id}",
|
||||
"id": value.id,
|
||||
"revision": value.revision,
|
||||
"key_hash": value.key_hash,
|
||||
"input_hash": value.input_hash,
|
||||
"action": value.action,
|
||||
"reason": value.reason,
|
||||
"correction": dict(value.correction) if value.correction else None,
|
||||
"actor_ref": value.actor_ref,
|
||||
"decided_at": value.decided_at,
|
||||
}
|
||||
|
||||
|
||||
def _source_name(name: str, identifier: str) -> str:
|
||||
slug = re.sub(r"[^a-z0-9]+", "_", name.casefold()).strip("_")
|
||||
return (f"decisions_{slug}" if slug else f"decisions_{identifier[:8]}")[:120]
|
||||
|
||||
|
||||
__all__: Sequence[str] = (
|
||||
"DECISION_SET_REF_PREFIX",
|
||||
"ReconciliationDecisionConflictError",
|
||||
"ReconciliationDecisionError",
|
||||
"ReconciliationDecisionNotFoundError",
|
||||
"create_decision_set",
|
||||
"current_decision_rows",
|
||||
"current_decisions",
|
||||
"decision_rows",
|
||||
"decision_set_fingerprint",
|
||||
"decision_set_id_from_ref",
|
||||
"decision_set_payload",
|
||||
"decision_set_source_payload",
|
||||
"get_decision_set",
|
||||
"list_decision_sets",
|
||||
"record_decision",
|
||||
)
|
||||
@@ -10,6 +10,7 @@ from govoplan_core.api.v1.schemas import (
|
||||
from govoplan_core.audit.logging import audit_event
|
||||
from govoplan_core.auth import ApiPrincipal, get_api_principal, has_scope
|
||||
from govoplan_core.core.automation import AutomationInvocation
|
||||
from govoplan_core.core.concurrency import RevisionConflictError
|
||||
from govoplan_core.core.dataflows import (
|
||||
DataflowPublicationTarget,
|
||||
DataflowRunRequest,
|
||||
@@ -31,6 +32,8 @@ from govoplan_core.core.references import (
|
||||
validate_access_scope_reference,
|
||||
)
|
||||
from govoplan_core.core.tabular_sources import (
|
||||
TabularCsvSource,
|
||||
TabularSourceError,
|
||||
parse_tabular_csv,
|
||||
)
|
||||
from govoplan_core.db.session import get_session
|
||||
@@ -67,10 +70,15 @@ from govoplan_dataflow.backend.schemas import (
|
||||
PipelineRunMetricsResponse,
|
||||
PipelineRunResponse,
|
||||
PipelinePromotionRequest,
|
||||
PipelineRebaseRequest,
|
||||
PipelineResponse,
|
||||
PipelineSqlResponse,
|
||||
PipelineUpdateRequest,
|
||||
PipelineValidationResponse,
|
||||
ReconciliationDecisionSetCreateRequest,
|
||||
ReconciliationDecisionSetListResponse,
|
||||
ReconciliationDecisionSetResponse,
|
||||
ReconciliationDecisionWriteRequest,
|
||||
TabularSnapshotCreateRequest,
|
||||
TabularSourceColumnResponse,
|
||||
TabularSourceListResponse,
|
||||
@@ -104,12 +112,24 @@ from govoplan_dataflow.backend.service import (
|
||||
pipeline_run_response,
|
||||
preview_pipeline,
|
||||
promote_pipeline,
|
||||
rebase_pipeline,
|
||||
render_graph_sql,
|
||||
start_pipeline_run,
|
||||
update_pipeline,
|
||||
validate_draft,
|
||||
)
|
||||
from govoplan_dataflow.backend.run_worker import run_metrics
|
||||
from govoplan_dataflow.backend.reconciliation_decisions import (
|
||||
ReconciliationDecisionConflictError,
|
||||
ReconciliationDecisionError,
|
||||
ReconciliationDecisionNotFoundError,
|
||||
create_decision_set,
|
||||
decision_set_payload,
|
||||
decision_set_source_payload,
|
||||
get_decision_set,
|
||||
list_decision_sets,
|
||||
record_decision,
|
||||
)
|
||||
from govoplan_dataflow.backend.recovery import dataflow_run_recovery_states
|
||||
from govoplan_dataflow.backend.triggers import (
|
||||
create_trigger,
|
||||
@@ -145,6 +165,18 @@ def _actor_id(principal: ApiPrincipal) -> str | None:
|
||||
)
|
||||
|
||||
|
||||
def _decision_actor_ref(principal: ApiPrincipal) -> str:
|
||||
actor_id = _actor_id(principal)
|
||||
if not actor_id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="A stable actor reference is required to record a decision.",
|
||||
)
|
||||
principal_ref = principal.to_platform_principal()
|
||||
prefix = "service-account" if principal_ref.service_account_id else "account"
|
||||
return f"{prefix}:{actor_id}"
|
||||
|
||||
|
||||
def _http_error(exc: DataflowError) -> HTTPException:
|
||||
if isinstance(exc, DataflowNotFoundError):
|
||||
return HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=str(exc))
|
||||
@@ -161,6 +193,17 @@ def _http_error(exc: DataflowError) -> HTTPException:
|
||||
return HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc))
|
||||
|
||||
|
||||
def _decision_http_error(exc: ReconciliationDecisionError) -> HTTPException:
|
||||
if isinstance(exc, ReconciliationDecisionNotFoundError):
|
||||
return HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=str(exc))
|
||||
if isinstance(exc, ReconciliationDecisionConflictError):
|
||||
return HTTPException(status_code=status.HTTP_409_CONFLICT, detail=str(exc))
|
||||
return HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail=str(exc),
|
||||
)
|
||||
|
||||
|
||||
def _governance_http_error(exc: PermissionError | ValueError) -> HTTPException:
|
||||
return HTTPException(
|
||||
status_code=(
|
||||
@@ -329,8 +372,8 @@ def api_list_sources(
|
||||
registry = get_registry()
|
||||
provider = datasource_catalogue(registry)
|
||||
writer = datasource_lifecycle(registry)
|
||||
if provider is None:
|
||||
return TabularSourceListResponse(available=False, writable=False, sources=[])
|
||||
sources = ()
|
||||
if provider is not None:
|
||||
try:
|
||||
sources = provider.list_datasources(
|
||||
session,
|
||||
@@ -340,10 +383,31 @@ def api_list_sources(
|
||||
)
|
||||
except DatasourceError as exc:
|
||||
raise _source_http_error(exc) from exc
|
||||
decision_sources = []
|
||||
for item in list_decision_sets(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
include_decisions=False,
|
||||
):
|
||||
if query and query.casefold() not in item.name.casefold():
|
||||
continue
|
||||
try:
|
||||
require_definition_action(
|
||||
item.pipeline,
|
||||
principal=principal,
|
||||
registry=registry,
|
||||
action="view",
|
||||
)
|
||||
except PermissionError:
|
||||
continue
|
||||
decision_sources.append(decision_set_source_payload(item))
|
||||
return TabularSourceListResponse(
|
||||
available=True,
|
||||
available=provider is not None or bool(decision_sources),
|
||||
writable=writer is not None,
|
||||
sources=[_source_response(source) for source in sources],
|
||||
sources=[
|
||||
*(_source_response(source) for source in sources),
|
||||
*(TabularSourceResponse.model_validate(item) for item in decision_sources),
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
@@ -370,6 +434,7 @@ def api_create_source_snapshot(
|
||||
payload.csv_text or "",
|
||||
delimiter=payload.delimiter,
|
||||
max_rows=10_000,
|
||||
value_mode=payload.csv_value_mode,
|
||||
)
|
||||
if payload.format == "csv"
|
||||
else tuple(payload.rows or ())
|
||||
@@ -386,6 +451,11 @@ def api_create_source_snapshot(
|
||||
shape="tabular",
|
||||
rows=rows,
|
||||
provider="dataflow.upload",
|
||||
csv_source=(TabularCsvSource(
|
||||
text=payload.csv_text or "",
|
||||
delimiter=payload.delimiter,
|
||||
value_mode=payload.csv_value_mode,
|
||||
) if payload.format == "csv" else None),
|
||||
provenance={
|
||||
"created_via": "dataflow",
|
||||
"source_format": payload.format,
|
||||
@@ -400,6 +470,8 @@ def api_create_source_snapshot(
|
||||
principal,
|
||||
stage_ref=stage.ref,
|
||||
)
|
||||
except TabularSourceError as exc:
|
||||
raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_CONTENT, detail=str(exc)) from exc
|
||||
except DatasourceError as exc:
|
||||
raise _source_http_error(exc) from exc
|
||||
audit_event(
|
||||
@@ -484,6 +556,8 @@ def api_create_pipeline(
|
||||
tenant_id=tenant_id or principal.tenant_id,
|
||||
actor_id=_actor_id(principal),
|
||||
payload=payload,
|
||||
principal=principal,
|
||||
registry=get_registry(),
|
||||
)
|
||||
except (PermissionError, ValueError) as exc:
|
||||
raise _governance_http_error(exc) from exc
|
||||
@@ -504,6 +578,212 @@ def api_create_pipeline(
|
||||
return response
|
||||
|
||||
|
||||
@router.get(
|
||||
"/pipelines/{pipeline_id}/decision-sets",
|
||||
response_model=ReconciliationDecisionSetListResponse,
|
||||
)
|
||||
def api_list_reconciliation_decision_sets(
|
||||
pipeline_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> ReconciliationDecisionSetListResponse:
|
||||
_require_any_scope(principal, READ_SCOPE, WRITE_SCOPE, RUN_SCOPE, ADMIN_SCOPE)
|
||||
try:
|
||||
pipeline = get_pipeline(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
pipeline_id=pipeline_id,
|
||||
)
|
||||
require_definition_action(
|
||||
pipeline,
|
||||
principal=principal,
|
||||
registry=get_registry(),
|
||||
action="view",
|
||||
)
|
||||
except PermissionError as exc:
|
||||
raise _governance_http_error(exc) from exc
|
||||
except DataflowError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
return ReconciliationDecisionSetListResponse(
|
||||
decision_sets=[
|
||||
ReconciliationDecisionSetResponse.model_validate(
|
||||
decision_set_payload(item, include_decisions=False)
|
||||
)
|
||||
for item in list_decision_sets(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
pipeline_id=pipeline_id,
|
||||
include_decisions=False,
|
||||
)
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/pipelines/{pipeline_id}/decision-sets",
|
||||
response_model=ReconciliationDecisionSetResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def api_create_reconciliation_decision_set(
|
||||
pipeline_id: str,
|
||||
payload: ReconciliationDecisionSetCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> ReconciliationDecisionSetResponse:
|
||||
_require_any_scope(principal, WRITE_SCOPE, ADMIN_SCOPE)
|
||||
try:
|
||||
pipeline = get_pipeline(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
pipeline_id=pipeline_id,
|
||||
)
|
||||
require_definition_action(
|
||||
pipeline,
|
||||
principal=principal,
|
||||
registry=get_registry(),
|
||||
action="edit",
|
||||
)
|
||||
item = create_decision_set(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
pipeline_id=pipeline_id,
|
||||
name=payload.name,
|
||||
node_id=payload.node_id,
|
||||
actor_ref=_decision_actor_ref(principal),
|
||||
)
|
||||
except PermissionError as exc:
|
||||
raise _governance_http_error(exc) from exc
|
||||
except DataflowError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
except ReconciliationDecisionError as exc:
|
||||
session.rollback()
|
||||
raise _decision_http_error(exc) from exc
|
||||
audit_event(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=getattr(principal.user, "id", None),
|
||||
api_key_id=principal.api_key_id,
|
||||
action="dataflow.reconciliation_decision_set.created",
|
||||
object_type="dataflow_reconciliation_decision_set",
|
||||
object_id=item.id,
|
||||
details={"pipeline_id": pipeline_id, "node_id": payload.node_id},
|
||||
)
|
||||
response = ReconciliationDecisionSetResponse.model_validate(
|
||||
decision_set_payload(item)
|
||||
)
|
||||
session.commit()
|
||||
return response
|
||||
|
||||
|
||||
@router.get(
|
||||
"/decision-sets/{decision_set_id}",
|
||||
response_model=ReconciliationDecisionSetResponse,
|
||||
)
|
||||
def api_get_reconciliation_decision_set(
|
||||
decision_set_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> ReconciliationDecisionSetResponse:
|
||||
_require_any_scope(principal, READ_SCOPE, WRITE_SCOPE, RUN_SCOPE, ADMIN_SCOPE)
|
||||
try:
|
||||
item = get_decision_set(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
decision_set_id=decision_set_id,
|
||||
)
|
||||
require_definition_action(
|
||||
item.pipeline,
|
||||
principal=principal,
|
||||
registry=get_registry(),
|
||||
action="view",
|
||||
)
|
||||
except PermissionError as exc:
|
||||
raise _governance_http_error(exc) from exc
|
||||
except ReconciliationDecisionError as exc:
|
||||
raise _decision_http_error(exc) from exc
|
||||
return ReconciliationDecisionSetResponse.model_validate(
|
||||
decision_set_payload(item)
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/decision-sets/{decision_set_id}/decisions",
|
||||
response_model=ReconciliationDecisionSetResponse,
|
||||
)
|
||||
def api_record_reconciliation_decision(
|
||||
decision_set_id: str,
|
||||
payload: ReconciliationDecisionWriteRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> ReconciliationDecisionSetResponse:
|
||||
_require_any_scope(principal, WRITE_SCOPE, ADMIN_SCOPE)
|
||||
try:
|
||||
item = get_decision_set(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
decision_set_id=decision_set_id,
|
||||
)
|
||||
pipeline = get_pipeline(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
pipeline_id=item.pipeline_id,
|
||||
)
|
||||
require_definition_action(
|
||||
pipeline,
|
||||
principal=principal,
|
||||
registry=get_registry(),
|
||||
action="edit",
|
||||
)
|
||||
item = record_decision(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
decision_set_id=decision_set_id,
|
||||
expected_revision=payload.base_revision,
|
||||
key_hash=payload.key_hash,
|
||||
input_hash=payload.input_hash,
|
||||
action=payload.action,
|
||||
reason=payload.reason,
|
||||
correction=payload.correction,
|
||||
actor_ref=_decision_actor_ref(principal),
|
||||
)
|
||||
except PermissionError as exc:
|
||||
raise _governance_http_error(exc) from exc
|
||||
except DataflowError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
except RevisionConflictError as exc:
|
||||
session.rollback()
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail=exc.as_dict(),
|
||||
) from exc
|
||||
except ReconciliationDecisionError as exc:
|
||||
session.rollback()
|
||||
raise _decision_http_error(exc) from exc
|
||||
decision = item.decisions[-1]
|
||||
audit_event(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=getattr(principal.user, "id", None),
|
||||
api_key_id=principal.api_key_id,
|
||||
action="dataflow.reconciliation_decision.recorded",
|
||||
object_type="dataflow_reconciliation_decision",
|
||||
object_id=decision.id,
|
||||
details={
|
||||
"decision_set_id": item.id,
|
||||
"pipeline_id": item.pipeline_id,
|
||||
"revision": decision.revision,
|
||||
"key_hash": decision.key_hash,
|
||||
"input_hash": decision.input_hash,
|
||||
"decision_action": decision.action,
|
||||
},
|
||||
)
|
||||
response = ReconciliationDecisionSetResponse.model_validate(
|
||||
decision_set_payload(item)
|
||||
)
|
||||
session.commit()
|
||||
return response
|
||||
|
||||
|
||||
@router.get("/pipelines/{pipeline_id}", response_model=PipelineResponse)
|
||||
def api_get_pipeline(
|
||||
pipeline_id: str,
|
||||
@@ -576,6 +856,8 @@ def api_update_pipeline(
|
||||
pipeline_id=pipeline_id,
|
||||
actor_id=_actor_id(principal),
|
||||
payload=payload,
|
||||
principal=principal,
|
||||
registry=get_registry(),
|
||||
)
|
||||
except (PermissionError, ValueError) as exc:
|
||||
raise _governance_http_error(exc) from exc
|
||||
@@ -702,6 +984,64 @@ def api_derive_pipeline(
|
||||
return response
|
||||
|
||||
|
||||
@router.post(
|
||||
"/pipelines/{pipeline_id}/rebase",
|
||||
response_model=PipelineResponse,
|
||||
)
|
||||
def api_rebase_pipeline(
|
||||
pipeline_id: str,
|
||||
payload: PipelineRebaseRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> PipelineResponse:
|
||||
_require_any_scope(principal, WRITE_SCOPE, ADMIN_SCOPE)
|
||||
try:
|
||||
existing = get_pipeline(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
pipeline_id=pipeline_id,
|
||||
)
|
||||
require_definition_action(
|
||||
existing,
|
||||
principal=principal,
|
||||
registry=get_registry(),
|
||||
action="edit",
|
||||
)
|
||||
pipeline = rebase_pipeline(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
pipeline_id=pipeline_id,
|
||||
actor_id=_actor_id(principal),
|
||||
principal=principal,
|
||||
registry=get_registry(),
|
||||
payload=payload,
|
||||
)
|
||||
except PermissionError as exc:
|
||||
raise _governance_http_error(exc) from exc
|
||||
except DataflowError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
audit_event(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=getattr(principal.user, "id", None),
|
||||
api_key_id=principal.api_key_id,
|
||||
action="dataflow.pipeline.rebased",
|
||||
object_type="dataflow_pipeline",
|
||||
object_id=pipeline.id,
|
||||
details={
|
||||
"child_revision": pipeline.current_revision,
|
||||
"source_pipeline_id": pipeline.derived_from_pipeline_id,
|
||||
"source_revision": pipeline.derived_from_revision,
|
||||
"source_hash": pipeline.derived_from_hash,
|
||||
"status": pipeline.status,
|
||||
"reason": payload.reason.strip(),
|
||||
},
|
||||
)
|
||||
response = _pipeline_response(session, pipeline, principal)
|
||||
session.commit()
|
||||
return response
|
||||
|
||||
|
||||
@router.get(
|
||||
"/pipelines/{pipeline_id}/triggers",
|
||||
response_model=DataflowTriggerListResponse,
|
||||
@@ -1237,10 +1577,22 @@ def api_promote_pipeline(
|
||||
@router.post("/validate", response_model=PipelineValidationResponse)
|
||||
def api_validate_pipeline(
|
||||
payload: PipelineDraftRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> PipelineValidationResponse:
|
||||
_require_any_scope(principal, READ_SCOPE, WRITE_SCOPE, RUN_SCOPE, ADMIN_SCOPE)
|
||||
return validate_draft(payload)
|
||||
try:
|
||||
return validate_draft(
|
||||
payload,
|
||||
session=session,
|
||||
tenant_id=principal.tenant_id,
|
||||
principal=principal,
|
||||
registry=get_registry(),
|
||||
)
|
||||
except PermissionError as exc:
|
||||
raise _governance_http_error(exc) from exc
|
||||
except DataflowError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post("/sql/compile", response_model=PipelineSqlResponse)
|
||||
@@ -1277,6 +1629,8 @@ def api_preview_pipeline(
|
||||
registry=get_registry(),
|
||||
payload=payload,
|
||||
)
|
||||
except PermissionError as exc:
|
||||
raise _governance_http_error(exc) from exc
|
||||
except DataflowError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
if response.pipeline_id:
|
||||
|
||||
@@ -129,7 +129,14 @@ def _propagation_context(
|
||||
def _inline_source(
|
||||
context: SchemaPropagationContext,
|
||||
) -> SchemaPropagationResult:
|
||||
return SchemaPropagationResult(_inline_schema(context.node.config.get("rows")))
|
||||
configured = _configured_schema(
|
||||
context.node.config.get("contract_schema")
|
||||
)
|
||||
return SchemaPropagationResult(
|
||||
configured
|
||||
if configured.columns
|
||||
else _inline_schema(context.node.config.get("rows"))
|
||||
)
|
||||
|
||||
|
||||
def _reference_source(
|
||||
@@ -724,16 +731,57 @@ def _comparison_columns(value: object) -> tuple[list[str], list[str]]:
|
||||
|
||||
|
||||
def _subflow(context: SchemaPropagationContext) -> SchemaPropagationResult:
|
||||
input_schema = _configured_schema(
|
||||
context.node.config.get("input_schema")
|
||||
)
|
||||
output_schema = _configured_schema(
|
||||
context.node.config.get("output_schema")
|
||||
)
|
||||
diagnostics: list[DataflowDiagnostic] = []
|
||||
if input_schema.columns and not context.input_state.open:
|
||||
missing = sorted(input_schema.columns - context.input_state.columns)
|
||||
if missing:
|
||||
diagnostics.append(
|
||||
_error(
|
||||
"subflow.input_contract.missing",
|
||||
"Subflow input is missing required contract columns: "
|
||||
+ ", ".join(missing),
|
||||
node_id=context.node.id,
|
||||
field="input_schema",
|
||||
)
|
||||
)
|
||||
incompatible = sorted(
|
||||
column
|
||||
for column in input_schema.columns & context.input_state.columns
|
||||
if not _compatible_contract_type(
|
||||
context.input_state.type_of(column),
|
||||
input_schema.type_of(column),
|
||||
)
|
||||
)
|
||||
if incompatible:
|
||||
diagnostics.append(
|
||||
_error(
|
||||
"subflow.input_contract.type",
|
||||
"Subflow input has incompatible contract types for: "
|
||||
+ ", ".join(incompatible),
|
||||
node_id=context.node.id,
|
||||
field="input_schema",
|
||||
)
|
||||
)
|
||||
return SchemaPropagationResult(
|
||||
output_schema
|
||||
if output_schema.columns
|
||||
else unknown_schema()
|
||||
else unknown_schema(),
|
||||
tuple(diagnostics),
|
||||
)
|
||||
|
||||
|
||||
def _compatible_contract_type(actual: str, expected: str) -> bool:
|
||||
if "unknown" in {actual, expected} or actual == expected:
|
||||
return True
|
||||
return {actual, expected} <= {"integer", "number"}
|
||||
|
||||
|
||||
def _identity(context: SchemaPropagationContext) -> SchemaPropagationResult:
|
||||
return SchemaPropagationResult(context.input_state)
|
||||
|
||||
|
||||
@@ -122,6 +122,11 @@ class PipelineGovernanceResponse(BaseModel):
|
||||
derived_from_pipeline_id: str | None
|
||||
derived_from_revision: int | None
|
||||
derived_from_hash: str | None
|
||||
source_available: bool = False
|
||||
source_name: str | None = None
|
||||
source_current_revision: int | None = None
|
||||
source_current_hash: str | None = None
|
||||
update_available: bool = False
|
||||
derivation_provenance: dict[str, Any] = Field(default_factory=dict)
|
||||
actions: dict[str, DefinitionActionDecisionResponse]
|
||||
|
||||
@@ -145,6 +150,68 @@ class PipelineListResponse(BaseModel):
|
||||
pipelines: list[PipelineResponse]
|
||||
|
||||
|
||||
class ReconciliationDecisionSetCreateRequest(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=300)
|
||||
node_id: str | None = Field(
|
||||
default=None,
|
||||
min_length=1,
|
||||
max_length=100,
|
||||
pattern=r"^[A-Za-z0-9_.:-]+$",
|
||||
)
|
||||
|
||||
|
||||
class ReconciliationDecisionWriteRequest(BaseModel):
|
||||
base_revision: int = Field(ge=1)
|
||||
key_hash: str = Field(pattern=r"^[0-9a-f]{64}$")
|
||||
input_hash: str = Field(pattern=r"^[0-9a-f]{64}$")
|
||||
action: Literal["accept", "reject", "correct", "defer"]
|
||||
reason: str = Field(min_length=3, max_length=4_000)
|
||||
correction: dict[str, Any] | None = None
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_correction(self) -> "ReconciliationDecisionWriteRequest":
|
||||
if self.action == "correct" and not self.correction:
|
||||
raise ValueError("Correct decisions require corrected field values.")
|
||||
if self.action != "correct" and self.correction:
|
||||
raise ValueError("Only correct decisions may include corrected field values.")
|
||||
return self
|
||||
|
||||
|
||||
class ReconciliationDecisionResponse(BaseModel):
|
||||
ref: str
|
||||
id: str
|
||||
revision: int
|
||||
key_hash: str
|
||||
input_hash: str
|
||||
action: Literal["accept", "reject", "correct", "defer"]
|
||||
reason: str
|
||||
correction: dict[str, Any] | None
|
||||
actor_ref: str
|
||||
decided_at: datetime
|
||||
|
||||
|
||||
class ReconciliationDecisionSetResponse(BaseModel):
|
||||
ref: str
|
||||
id: str
|
||||
pipeline_id: str
|
||||
name: str
|
||||
node_id: str | None
|
||||
resource_revision: int = Field(ge=1)
|
||||
etag: str
|
||||
fingerprint: str
|
||||
decisions_included: bool
|
||||
current_decisions: list[ReconciliationDecisionResponse]
|
||||
history: list[ReconciliationDecisionResponse]
|
||||
created_by: str | None
|
||||
updated_by: str | None
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
|
||||
class ReconciliationDecisionSetListResponse(BaseModel):
|
||||
decision_sets: list[ReconciliationDecisionSetResponse]
|
||||
|
||||
|
||||
class PipelineCreateRequest(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=300)
|
||||
description: str | None = Field(default=None, max_length=4000)
|
||||
@@ -184,7 +251,23 @@ class PipelineDeriveRequest(BaseModel):
|
||||
allow_automation: bool = False
|
||||
|
||||
|
||||
class PipelineRebaseRequest(BaseModel):
|
||||
expected_revision: int = Field(ge=1)
|
||||
source_revision: int = Field(ge=1)
|
||||
source_hash: str = Field(pattern=r"^[0-9a-f]{64}$")
|
||||
reason: str = Field(min_length=3, max_length=4_000)
|
||||
|
||||
@field_validator("reason")
|
||||
@classmethod
|
||||
def validate_reason(cls, value: str) -> str:
|
||||
cleaned = value.strip()
|
||||
if len(cleaned) < 3:
|
||||
raise ValueError("A meaningful rebase review reason is required.")
|
||||
return cleaned
|
||||
|
||||
|
||||
class PipelineDraftRequest(BaseModel):
|
||||
pipeline_id: str | None = Field(default=None, max_length=36)
|
||||
graph: PipelineGraph | None = None
|
||||
sql_text: str | None = Field(default=None, max_length=100_000)
|
||||
source_nodes: list[GraphNode] = Field(default_factory=list, max_length=20)
|
||||
@@ -654,6 +737,7 @@ class TabularSnapshotCreateRequest(BaseModel):
|
||||
format: Literal["json", "csv"] = "json"
|
||||
rows: list[dict[str, Any]] | None = Field(default=None, max_length=10_000)
|
||||
csv_text: str | None = Field(default=None, max_length=5_000_000)
|
||||
csv_value_mode: Literal["legacy_typed", "text"] = "legacy_typed"
|
||||
delimiter: str = Field(default=",", min_length=1, max_length=1)
|
||||
|
||||
@model_validator(mode="after")
|
||||
|
||||
@@ -28,6 +28,7 @@ from govoplan_core.core.datasources import (
|
||||
datasource_publication,
|
||||
)
|
||||
from govoplan_core.db.base import utcnow
|
||||
from govoplan_core.security.worker_payload import WorkerPayloadError, encode_worker_payload
|
||||
from govoplan_dataflow.backend.backends import (
|
||||
BackendExecutionError,
|
||||
BackendSource,
|
||||
@@ -35,6 +36,7 @@ from govoplan_dataflow.backend.backends import (
|
||||
execute_typed_graph,
|
||||
)
|
||||
from govoplan_dataflow.backend.batches import TypedBatch
|
||||
from govoplan_dataflow.backend.backends.reference import reference_source_key
|
||||
from govoplan_dataflow.backend.db.models import (
|
||||
DataflowPipeline,
|
||||
DataflowPipelineDeployment,
|
||||
@@ -48,7 +50,6 @@ from govoplan_dataflow.backend.executor import (
|
||||
PipelineExecutionError,
|
||||
PipelineExecutionResult,
|
||||
ResolvedSource,
|
||||
execute_preview,
|
||||
)
|
||||
from govoplan_dataflow.backend.governance import (
|
||||
definition_governance_payload,
|
||||
@@ -60,6 +61,7 @@ from govoplan_dataflow.backend.graph import (
|
||||
preserve_compatible_graph_layout,
|
||||
validate_graph,
|
||||
)
|
||||
from govoplan_dataflow.backend.ir import graph_to_ir
|
||||
from govoplan_dataflow.backend.schemas import (
|
||||
DataflowDiagnostic,
|
||||
GraphNode,
|
||||
@@ -72,6 +74,7 @@ from govoplan_dataflow.backend.schemas import (
|
||||
PipelinePreviewResponse,
|
||||
PipelineDeploymentResponse,
|
||||
PipelinePromotionRequest,
|
||||
PipelineRebaseRequest,
|
||||
PipelineResponse,
|
||||
PipelineRevisionResponse,
|
||||
PipelineRunResponse,
|
||||
@@ -86,6 +89,12 @@ from govoplan_dataflow.backend.recovery import (
|
||||
begin_dataflow_run_recovery,
|
||||
dataflow_run_recovery_state,
|
||||
)
|
||||
from govoplan_dataflow.backend.reconciliation_decisions import (
|
||||
current_decision_rows,
|
||||
decision_set_fingerprint,
|
||||
decision_set_id_from_ref,
|
||||
get_decision_set,
|
||||
)
|
||||
from govoplan_dataflow.backend.sql_compiler import (
|
||||
SqlCompilationError,
|
||||
compile_sql,
|
||||
@@ -181,15 +190,177 @@ def get_pipeline_revision(
|
||||
return item
|
||||
|
||||
|
||||
def _resolve_reusable_subflows(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
graph: PipelineGraph,
|
||||
principal: ApiPrincipal | None,
|
||||
registry: object | None,
|
||||
target_pipeline_id: str | None,
|
||||
ancestry: tuple[str, ...] = (),
|
||||
) -> PipelineGraph:
|
||||
if not any(node.type == "subflow" for node in graph.nodes):
|
||||
return graph
|
||||
if principal is None:
|
||||
raise DataflowConflictError(
|
||||
"Reusable subflows require a tenant principal and current Policy "
|
||||
"decision."
|
||||
)
|
||||
resolved_nodes: list[GraphNode] = []
|
||||
for node in graph.nodes:
|
||||
if node.type != "subflow":
|
||||
resolved_nodes.append(node)
|
||||
continue
|
||||
source_id = _pipeline_id_from_ref(node.config.get("template_ref"))
|
||||
if source_id == target_pipeline_id:
|
||||
raise DataflowConflictError(
|
||||
"A pipeline cannot reference itself as a reusable subflow."
|
||||
)
|
||||
source = get_pipeline(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
pipeline_id=source_id,
|
||||
)
|
||||
reuse_decision = require_definition_action(
|
||||
source,
|
||||
principal=principal,
|
||||
registry=registry,
|
||||
action="reuse",
|
||||
)
|
||||
source_revision_number = _subflow_revision(
|
||||
node.config.get("template_version")
|
||||
)
|
||||
source_revision = get_pipeline_revision(
|
||||
session,
|
||||
pipeline=source,
|
||||
revision=source_revision_number,
|
||||
)
|
||||
reference_key = f"{source.id}:{source_revision.revision}"
|
||||
if reference_key in ancestry:
|
||||
raise DataflowConflictError(
|
||||
"Reusable subflow references contain a cycle at "
|
||||
f"pipeline:{source.id} revision {source_revision.revision}."
|
||||
)
|
||||
nested = _resolve_reusable_subflows(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
graph=PipelineGraph.model_validate(source_revision.graph),
|
||||
principal=principal,
|
||||
registry=registry,
|
||||
target_pipeline_id=target_pipeline_id,
|
||||
ancestry=(*ancestry, reference_key),
|
||||
)
|
||||
input_nodes = [
|
||||
item
|
||||
for item in nested.nodes
|
||||
if item.type == "source.inline"
|
||||
and item.config.get("input_binding") is True
|
||||
]
|
||||
if len(input_nodes) != 1:
|
||||
raise DataflowConflictError(
|
||||
"A referenced reusable definition must declare exactly one "
|
||||
"inline template input binding."
|
||||
)
|
||||
typed = graph_to_ir(nested)
|
||||
typed_by_id = {item.id: item for item in typed.nodes}
|
||||
output_nodes = [item for item in nested.nodes if item.type == "output"]
|
||||
if len(output_nodes) != 1:
|
||||
raise DataflowConflictError(
|
||||
"A referenced reusable definition must have exactly one "
|
||||
"typed output."
|
||||
)
|
||||
input_contract = _typed_contract(
|
||||
typed_by_id[input_nodes[0].id].output_schema,
|
||||
label="input",
|
||||
)
|
||||
output_contract = _typed_contract(
|
||||
typed_by_id[output_nodes[0].id].output_schema,
|
||||
label="output",
|
||||
)
|
||||
config = {
|
||||
**node.config,
|
||||
"template_ref": f"pipeline:{source.id}",
|
||||
"template_version": str(source_revision.revision),
|
||||
"template_hash": source_revision.content_hash,
|
||||
"graph": canonical_graph_payload(nested),
|
||||
"input_schema": input_contract,
|
||||
"output_schema": output_contract,
|
||||
"reference_provenance": {
|
||||
"source_scope": {
|
||||
"scope_type": source.scope_type,
|
||||
"scope_id": source.scope_id,
|
||||
},
|
||||
"source_definition_kind": source.definition_kind,
|
||||
"policy_decision": reuse_decision.to_dict(),
|
||||
},
|
||||
}
|
||||
resolved_nodes.append(
|
||||
node.model_copy(update={"config": config}, deep=True)
|
||||
)
|
||||
return graph.model_copy(update={"nodes": resolved_nodes}, deep=True)
|
||||
|
||||
|
||||
def _pipeline_id_from_ref(value: object) -> str:
|
||||
text = str(value or "").strip()
|
||||
if not text.startswith("pipeline:") or len(text) <= len("pipeline:"):
|
||||
raise DataflowConflictError(
|
||||
"Reusable subflows require a canonical pipeline reference."
|
||||
)
|
||||
return text.removeprefix("pipeline:")
|
||||
|
||||
|
||||
def _subflow_revision(value: object) -> int:
|
||||
try:
|
||||
revision = int(str(value).strip())
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise DataflowConflictError(
|
||||
"Reusable subflows require a valid immutable source revision."
|
||||
) from exc
|
||||
if revision < 1:
|
||||
raise DataflowConflictError(
|
||||
"Reusable subflow revisions must be positive."
|
||||
)
|
||||
return revision
|
||||
|
||||
|
||||
def _typed_contract(schema: object, *, label: str) -> list[dict[str, object]]:
|
||||
fields = tuple(getattr(schema, "fields", ()))
|
||||
if not fields or any(getattr(item, "type", "unknown") == "unknown" for item in fields):
|
||||
raise DataflowConflictError(
|
||||
f"The reusable definition needs a closed typed {label} contract. "
|
||||
"Provide representative typed rows at its template input."
|
||||
)
|
||||
return [
|
||||
{
|
||||
"name": str(item.name),
|
||||
"type": str(item.type),
|
||||
"nullable": bool(item.nullable),
|
||||
}
|
||||
for item in fields
|
||||
]
|
||||
|
||||
|
||||
def create_pipeline(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
actor_id: str | None,
|
||||
payload: PipelineCreateRequest,
|
||||
principal: ApiPrincipal | None = None,
|
||||
registry: object | None = None,
|
||||
) -> DataflowPipeline:
|
||||
definition = normalize_definition(
|
||||
pipeline_id = new_uuid()
|
||||
graph = _resolve_reusable_subflows(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
graph=payload.graph,
|
||||
principal=principal,
|
||||
registry=registry,
|
||||
target_pipeline_id=pipeline_id,
|
||||
)
|
||||
definition = normalize_definition(
|
||||
graph=graph,
|
||||
sql_text=payload.sql_text,
|
||||
editor_mode=payload.editor_mode,
|
||||
)
|
||||
@@ -203,6 +374,7 @@ def create_pipeline(
|
||||
else payload.scope_id
|
||||
)
|
||||
pipeline = DataflowPipeline(
|
||||
id=pipeline_id,
|
||||
tenant_id=stored_tenant_id,
|
||||
scope_type=payload.scope_type,
|
||||
scope_id=scope_id,
|
||||
@@ -242,6 +414,8 @@ def update_pipeline(
|
||||
pipeline_id: str,
|
||||
actor_id: str | None,
|
||||
payload: PipelineUpdateRequest,
|
||||
principal: ApiPrincipal | None = None,
|
||||
registry: object | None = None,
|
||||
) -> DataflowPipeline:
|
||||
pipeline = get_pipeline(session, tenant_id=tenant_id, pipeline_id=pipeline_id)
|
||||
if payload.expected_revision != pipeline.current_revision:
|
||||
@@ -264,8 +438,16 @@ def update_pipeline(
|
||||
raise DataflowConflictError(
|
||||
"Definition kind is immutable; derive a flow or template instead."
|
||||
)
|
||||
definition = normalize_definition(
|
||||
graph = _resolve_reusable_subflows(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
graph=payload.graph,
|
||||
principal=principal,
|
||||
registry=registry,
|
||||
target_pipeline_id=pipeline.id,
|
||||
)
|
||||
definition = normalize_definition(
|
||||
graph=graph,
|
||||
sql_text=payload.sql_text,
|
||||
editor_mode=payload.editor_mode,
|
||||
)
|
||||
@@ -411,6 +593,195 @@ def derive_pipeline(
|
||||
return pipeline
|
||||
|
||||
|
||||
def pipeline_source_update_status(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
pipeline: DataflowPipeline,
|
||||
) -> dict[str, object]:
|
||||
source_id = pipeline.derived_from_pipeline_id
|
||||
if not source_id:
|
||||
return {
|
||||
"source_available": False,
|
||||
"source_name": None,
|
||||
"source_current_revision": None,
|
||||
"source_current_hash": None,
|
||||
"update_available": False,
|
||||
}
|
||||
source = session.scalar(
|
||||
select(DataflowPipeline).where(
|
||||
DataflowPipeline.id == source_id,
|
||||
or_(
|
||||
DataflowPipeline.tenant_id == tenant_id,
|
||||
DataflowPipeline.tenant_id.is_(None),
|
||||
),
|
||||
DataflowPipeline.deleted_at.is_(None),
|
||||
)
|
||||
)
|
||||
if source is None:
|
||||
return {
|
||||
"source_available": False,
|
||||
"source_name": None,
|
||||
"source_current_revision": None,
|
||||
"source_current_hash": None,
|
||||
"update_available": False,
|
||||
}
|
||||
revision = get_pipeline_revision(session, pipeline=source)
|
||||
return {
|
||||
"source_available": True,
|
||||
"source_name": source.name,
|
||||
"source_current_revision": revision.revision,
|
||||
"source_current_hash": revision.content_hash,
|
||||
"update_available": (
|
||||
revision.revision != pipeline.derived_from_revision
|
||||
or revision.content_hash != pipeline.derived_from_hash
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def rebase_pipeline(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
pipeline_id: str,
|
||||
actor_id: str | None,
|
||||
principal: ApiPrincipal,
|
||||
registry: object | None,
|
||||
payload: PipelineRebaseRequest,
|
||||
) -> DataflowPipeline:
|
||||
pipeline = get_pipeline(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
pipeline_id=pipeline_id,
|
||||
)
|
||||
if payload.expected_revision != pipeline.current_revision:
|
||||
raise DataflowConflictError(
|
||||
"Derived pipeline changed on the server; expected revision "
|
||||
f"{payload.expected_revision}, current revision is "
|
||||
f"{pipeline.current_revision}."
|
||||
)
|
||||
source_id = pipeline.derived_from_pipeline_id
|
||||
if not source_id:
|
||||
raise DataflowConflictError(
|
||||
"Only a pipeline derived from another definition can adopt a "
|
||||
"source update."
|
||||
)
|
||||
source = get_pipeline(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
pipeline_id=source_id,
|
||||
)
|
||||
reuse_decision = require_definition_action(
|
||||
source,
|
||||
principal=principal,
|
||||
registry=registry,
|
||||
action="derive",
|
||||
)
|
||||
source_revision = get_pipeline_revision(
|
||||
session,
|
||||
pipeline=source,
|
||||
revision=payload.source_revision,
|
||||
)
|
||||
if source_revision.content_hash != payload.source_hash:
|
||||
raise DataflowConflictError(
|
||||
"The reviewed source hash no longer matches the requested "
|
||||
"revision; reload before adopting the update."
|
||||
)
|
||||
if (
|
||||
pipeline.derived_from_revision is not None
|
||||
and source_revision.revision <= pipeline.derived_from_revision
|
||||
):
|
||||
raise DataflowConflictError(
|
||||
"A source update must use a revision newer than the currently "
|
||||
"pinned revision."
|
||||
)
|
||||
|
||||
previous_child_revision = pipeline.current_revision
|
||||
previous_child = get_pipeline_revision(session, pipeline=pipeline)
|
||||
previous_source_revision = pipeline.derived_from_revision
|
||||
previous_source_hash = pipeline.derived_from_hash
|
||||
source_limits = _effective_governance_limits(
|
||||
source,
|
||||
decision_details=reuse_decision.details,
|
||||
)
|
||||
effective_limits = {
|
||||
"inherit_to_lower_scopes": (
|
||||
pipeline.inherit_to_lower_scopes
|
||||
and source_limits["inherit_to_lower_scopes"]
|
||||
),
|
||||
"allow_run": pipeline.allow_run and source_limits["allow_run"],
|
||||
"allow_reuse": pipeline.allow_reuse and source_limits["allow_reuse"],
|
||||
"allow_automation": (
|
||||
pipeline.allow_automation and source_limits["allow_automation"]
|
||||
),
|
||||
}
|
||||
next_child_revision = previous_child_revision + 1
|
||||
rebased_at = utcnow()
|
||||
history_value = pipeline.derivation_provenance.get("rebase_history", [])
|
||||
history = list(history_value) if isinstance(history_value, list) else []
|
||||
history.append(
|
||||
{
|
||||
"child_revision_before": previous_child_revision,
|
||||
"child_hash_before": previous_child.content_hash,
|
||||
"child_revision_after": next_child_revision,
|
||||
"source_revision_before": previous_source_revision,
|
||||
"source_hash_before": previous_source_hash,
|
||||
"source_revision_after": source_revision.revision,
|
||||
"source_hash_after": source_revision.content_hash,
|
||||
"policy_decision": reuse_decision.to_dict(),
|
||||
"reason": payload.reason.strip(),
|
||||
"rebased_by": actor_id,
|
||||
"rebased_at": rebased_at.isoformat(),
|
||||
}
|
||||
)
|
||||
provenance = dict(pipeline.derivation_provenance)
|
||||
provenance.update(
|
||||
{
|
||||
"source_ref": f"pipeline:{source.id}",
|
||||
"source_scope": {
|
||||
"scope_type": source.scope_type,
|
||||
"scope_id": source.scope_id,
|
||||
},
|
||||
"source_definition_kind": source.definition_kind,
|
||||
"source_revision": source_revision.revision,
|
||||
"source_hash": source_revision.content_hash,
|
||||
"source_effective_limits": effective_limits,
|
||||
"policy_decision": reuse_decision.to_dict(),
|
||||
"last_rebased_by": actor_id,
|
||||
"last_rebased_at": rebased_at.isoformat(),
|
||||
"last_rebase_reason": payload.reason.strip(),
|
||||
"rebase_history": history,
|
||||
}
|
||||
)
|
||||
|
||||
pipeline.current_revision = next_child_revision
|
||||
pipeline.status = "draft"
|
||||
pipeline.inherit_to_lower_scopes = effective_limits[
|
||||
"inherit_to_lower_scopes"
|
||||
]
|
||||
pipeline.allow_run = effective_limits["allow_run"]
|
||||
pipeline.allow_reuse = effective_limits["allow_reuse"]
|
||||
pipeline.allow_automation = effective_limits["allow_automation"]
|
||||
pipeline.derived_from_revision = source_revision.revision
|
||||
pipeline.derived_from_hash = source_revision.content_hash
|
||||
pipeline.derivation_provenance = provenance
|
||||
pipeline.updated_by = actor_id
|
||||
pipeline.revisions.append(
|
||||
DataflowPipelineRevision(
|
||||
tenant_id=pipeline.tenant_id,
|
||||
revision=next_child_revision,
|
||||
schema_version=source_revision.schema_version,
|
||||
graph=json.loads(json.dumps(source_revision.graph)),
|
||||
sql_text=source_revision.sql_text,
|
||||
editor_mode=source_revision.editor_mode,
|
||||
content_hash=source_revision.content_hash,
|
||||
created_by=actor_id,
|
||||
)
|
||||
)
|
||||
session.flush()
|
||||
return pipeline
|
||||
|
||||
|
||||
def delete_pipeline(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -449,11 +820,36 @@ def pipeline_response(
|
||||
pipeline,
|
||||
principal=principal,
|
||||
registry=registry,
|
||||
source_update=pipeline_source_update_status(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
pipeline=pipeline,
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def validate_draft(payload: PipelineDraftRequest) -> PipelineValidationResponse:
|
||||
def validate_draft(
|
||||
payload: PipelineDraftRequest,
|
||||
*,
|
||||
session: Session | None = None,
|
||||
tenant_id: str | None = None,
|
||||
principal: ApiPrincipal | None = None,
|
||||
registry: object | None = None,
|
||||
) -> PipelineValidationResponse:
|
||||
if payload.graph is not None and session is not None and tenant_id is not None:
|
||||
payload = payload.model_copy(
|
||||
update={
|
||||
"graph": _resolve_reusable_subflows(
|
||||
session,
|
||||
tenant_id=tenant_id,
|
||||
graph=payload.graph,
|
||||
principal=principal,
|
||||
registry=registry,
|
||||
target_pipeline_id=payload.pipeline_id,
|
||||
)
|
||||
}
|
||||
)
|
||||
if payload.sql_text and payload.sql_text.strip():
|
||||
try:
|
||||
graph, sql_text, diagnostics = compile_sql(
|
||||
@@ -592,7 +988,13 @@ def preview_pipeline(
|
||||
sql_text=payload.sql_text,
|
||||
source_nodes=payload.source_nodes,
|
||||
)
|
||||
validated = validate_draft(draft)
|
||||
validated = validate_draft(
|
||||
draft,
|
||||
session=session,
|
||||
tenant_id=tenant_id,
|
||||
principal=principal,
|
||||
registry=registry,
|
||||
)
|
||||
if not validated.valid or validated.graph is None:
|
||||
return PipelinePreviewResponse(
|
||||
run_id=None,
|
||||
@@ -721,20 +1123,11 @@ def _execute_pipeline_preview(
|
||||
principal=principal,
|
||||
registry=registry,
|
||||
)
|
||||
if backend == "reference":
|
||||
return (
|
||||
execute_preview(
|
||||
graph,
|
||||
row_limit=row_limit,
|
||||
source_resolver=source_resolver,
|
||||
preview_node_id=preview_node_id,
|
||||
),
|
||||
EXECUTOR_VERSION,
|
||||
)
|
||||
sources = _typed_backend_sources(
|
||||
graph,
|
||||
source_resolver=source_resolver,
|
||||
source_limit=max(MAX_SOURCE_ROWS, row_limit),
|
||||
source_limit=MAX_SOURCE_ROWS if backend == "reference" else max(MAX_SOURCE_ROWS, row_limit),
|
||||
include_subflows=backend == "reference",
|
||||
)
|
||||
try:
|
||||
result = execute_typed_graph(
|
||||
@@ -748,8 +1141,14 @@ def _execute_pipeline_preview(
|
||||
raise PipelineExecutionError(
|
||||
str(exc),
|
||||
node_id=exc.node_id,
|
||||
diagnostics=tuple(exc.diagnostics),
|
||||
retryable=exc.code == "backend.capacity",
|
||||
diagnostics=(*exc.diagnostics, DataflowDiagnostic(
|
||||
severity="error", code=exc.code, message=str(exc), node_id=exc.node_id,
|
||||
)),
|
||||
node_diagnostics=exc.node_diagnostics,
|
||||
source_fingerprints=exc.source_fingerprints,
|
||||
input_row_count=exc.input_row_count,
|
||||
node_preview=exc.node_preview,
|
||||
retryable=exc.code in {"backend.capacity", "backend.process.busy"},
|
||||
) from exc
|
||||
columns = [
|
||||
PreviewColumn(
|
||||
@@ -826,13 +1225,51 @@ def _typed_backend_sources(
|
||||
*,
|
||||
source_resolver,
|
||||
source_limit: int = MAX_SOURCE_ROWS,
|
||||
include_subflows: bool = False,
|
||||
_depth: int = 0,
|
||||
_remaining_source_bytes: list[int] | None = None,
|
||||
) -> dict[str, BackendSource]:
|
||||
if _depth > 5:
|
||||
raise PipelineExecutionError("Subflows are limited to five nested levels.")
|
||||
if _remaining_source_bytes is None:
|
||||
_remaining_source_bytes = [32 * 1024 * 1024]
|
||||
sources: dict[str, BackendSource] = {}
|
||||
for node in graph.nodes:
|
||||
if include_subflows and node.type == "subflow":
|
||||
from govoplan_dataflow.backend.subflows import substitute_parameters
|
||||
|
||||
parameters = node.config.get("parameters")
|
||||
nested = PipelineGraph.model_validate(substitute_parameters(
|
||||
node.config.get("graph"), parameters if isinstance(parameters, dict) else {},
|
||||
))
|
||||
sources.update(_typed_backend_sources(
|
||||
nested, source_resolver=source_resolver, source_limit=source_limit,
|
||||
include_subflows=True, _depth=_depth + 1,
|
||||
_remaining_source_bytes=_remaining_source_bytes,
|
||||
))
|
||||
if node.type != "source.reference":
|
||||
continue
|
||||
if include_subflows and _remaining_source_bytes[0] <= 0:
|
||||
raise PipelineExecutionError(
|
||||
"Combined source data exceeds the 32 MiB transfer budget.", node_id=node.id,
|
||||
)
|
||||
resolved = source_resolver(node, source_limit)
|
||||
sources[node.id] = BackendSource(
|
||||
if include_subflows:
|
||||
try:
|
||||
# Check before constructing another columnar copy. The provider
|
||||
# still owns bounds on its individual authorized read; do not keep
|
||||
# accumulating individually valid batches before the worker gate.
|
||||
encoded_size = len(encode_worker_payload(
|
||||
tuple(dict(row) for row in resolved.rows), max_bytes=_remaining_source_bytes[0],
|
||||
))
|
||||
except WorkerPayloadError as exc:
|
||||
raise PipelineExecutionError(
|
||||
"Combined source data exceeds the 32 MiB transfer budget or contains unsupported values.",
|
||||
node_id=node.id,
|
||||
) from exc
|
||||
_remaining_source_bytes[0] -= encoded_size
|
||||
key = reference_source_key(node) if include_subflows else node.id
|
||||
sources[key] = BackendSource(
|
||||
node_id=node.id,
|
||||
batch=TypedBatch.from_rows(resolved.rows),
|
||||
source_ref=resolved.source_ref,
|
||||
@@ -1352,8 +1789,12 @@ def _run_authorization_payload(
|
||||
graph = PipelineGraph.model_validate(revision.graph)
|
||||
scopes = {RUN_SCOPE}
|
||||
if any(
|
||||
(
|
||||
node.type == "source.reference"
|
||||
or bool(node.config.get("source_ref"))
|
||||
)
|
||||
and decision_set_id_from_ref(str(node.config.get("source_ref") or ""))
|
||||
is None
|
||||
for node in graph.nodes
|
||||
):
|
||||
scopes.add(DATASOURCE_READ_SCOPE)
|
||||
@@ -1574,6 +2015,26 @@ def _publish_pipeline_result(
|
||||
run.output_publication_ref = publication.ref
|
||||
run.output_datasource_ref = publication.datasource.ref
|
||||
run.output_materialization_ref = publication.materialization.ref
|
||||
if publication.status in {"published_with_warnings", "review_required"}:
|
||||
diagnostics = list(run.diagnostics)
|
||||
diagnostics.append(
|
||||
DataflowDiagnostic(
|
||||
severity="warning",
|
||||
code=(
|
||||
"publication.review_required"
|
||||
if publication.status == "review_required"
|
||||
else "publication.warning"
|
||||
),
|
||||
message=(
|
||||
"The output materialization requires review before it can "
|
||||
"become the datasource's current state."
|
||||
if publication.status == "review_required"
|
||||
else "The output was published with datasource validation "
|
||||
"warnings."
|
||||
),
|
||||
).model_dump(mode="json")
|
||||
)
|
||||
run.diagnostics = diagnostics
|
||||
|
||||
|
||||
def _mark_pipeline_run_failed(
|
||||
@@ -1880,6 +2341,49 @@ def _datasource_source_resolver(
|
||||
provider = datasource_catalogue(registry)
|
||||
|
||||
def resolve_source(node: GraphNode, limit: int) -> ResolvedSource:
|
||||
source_ref = str(node.config.get("source_ref") or "")
|
||||
decision_set_id = decision_set_id_from_ref(source_ref)
|
||||
if decision_set_id is not None:
|
||||
try:
|
||||
decision_set = get_decision_set(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
decision_set_id=decision_set_id,
|
||||
include_decisions=False,
|
||||
)
|
||||
require_definition_action(
|
||||
decision_set.pipeline,
|
||||
principal=principal,
|
||||
registry=registry,
|
||||
action="view",
|
||||
)
|
||||
except (PermissionError, ValueError) as exc:
|
||||
raise PipelineExecutionError(
|
||||
str(exc),
|
||||
node_id=node.id,
|
||||
) from exc
|
||||
rows, total_rows = current_decision_rows(
|
||||
session,
|
||||
decision_set_id=decision_set.id,
|
||||
limit=limit,
|
||||
)
|
||||
fingerprint = decision_set_fingerprint(decision_set)
|
||||
expected_fingerprint = _clean_optional(
|
||||
node.config.get("expected_fingerprint")
|
||||
)
|
||||
if expected_fingerprint and expected_fingerprint != fingerprint:
|
||||
raise PipelineExecutionError(
|
||||
"Reconciliation decisions changed; refresh the decision source before running it.",
|
||||
node_id=node.id,
|
||||
)
|
||||
return ResolvedSource(
|
||||
rows=rows,
|
||||
source_ref=source_ref,
|
||||
provider="dataflow.reconciliation_decisions",
|
||||
fingerprint=fingerprint,
|
||||
total_rows=total_rows,
|
||||
truncated=total_rows > len(rows),
|
||||
)
|
||||
if provider is None:
|
||||
raise PipelineExecutionError(
|
||||
"Datasource-backed execution requires the Datasources "
|
||||
@@ -2217,11 +2721,13 @@ __all__ = [
|
||||
"list_pipelines",
|
||||
"normalize_definition",
|
||||
"pipeline_response",
|
||||
"pipeline_source_update_status",
|
||||
"pipeline_deployment_response",
|
||||
"pipeline_run_descriptor",
|
||||
"pipeline_run_request",
|
||||
"pipeline_run_response",
|
||||
"promote_pipeline",
|
||||
"rebase_pipeline",
|
||||
"preview_pipeline",
|
||||
"render_graph_sql",
|
||||
"start_pipeline_run",
|
||||
|
||||
Executable
+121
@@ -0,0 +1,121 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
from fastapi import HTTPException
|
||||
from pydantic import ValidationError
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.core.access import PrincipalRef
|
||||
from govoplan_dataflow.backend.router import WRITE_SCOPE, api_create_source_snapshot
|
||||
from govoplan_dataflow.backend.schemas import TabularSnapshotCreateRequest
|
||||
|
||||
|
||||
class CsvStagingRouteTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.principal = ApiPrincipal(
|
||||
principal=PrincipalRef(
|
||||
account_id="account",
|
||||
membership_id="membership",
|
||||
tenant_id="tenant",
|
||||
scopes=frozenset({WRITE_SCOPE}),
|
||||
),
|
||||
user=object(),
|
||||
account=object(),
|
||||
)
|
||||
self.session = Mock()
|
||||
self.writer = Mock()
|
||||
self.writer.create_stage.return_value = SimpleNamespace(ref="stage:one")
|
||||
self.writer.promote_stage.return_value = (
|
||||
SimpleNamespace(
|
||||
ref="datasource:one",
|
||||
provider="dataflow.upload",
|
||||
source_name="upload",
|
||||
row_count=1,
|
||||
fingerprint="f" * 64,
|
||||
),
|
||||
SimpleNamespace(ref="materialization:one"),
|
||||
)
|
||||
for name, options in (
|
||||
("get_registry", {"return_value": object()}),
|
||||
("datasource_lifecycle", {"return_value": self.writer}),
|
||||
("audit_event", {}),
|
||||
("_source_response", {"side_effect": lambda value: value}),
|
||||
):
|
||||
active = patch(f"govoplan_dataflow.backend.router.{name}", **options)
|
||||
active.start()
|
||||
self.addCleanup(active.stop)
|
||||
|
||||
def test_csv_stage_receives_exact_original_and_selected_parser_mode(self) -> None:
|
||||
text = 'value\r\n" 001 "\r\n'
|
||||
for mode, expected in (("text", " 001 "), ("legacy_typed", "001")):
|
||||
with self.subTest(mode=mode):
|
||||
api_create_source_snapshot(
|
||||
TabularSnapshotCreateRequest(
|
||||
name="Upload",
|
||||
source_name="upload",
|
||||
format="csv",
|
||||
csv_text=text,
|
||||
csv_value_mode=mode,
|
||||
),
|
||||
session=self.session,
|
||||
principal=self.principal,
|
||||
)
|
||||
stage = self.writer.create_stage.call_args.kwargs["stage"]
|
||||
self.assertEqual(({"value": expected},), stage.rows)
|
||||
self.assertEqual(text, stage.csv_source.text)
|
||||
self.assertEqual(mode, stage.csv_source.value_mode)
|
||||
self.assertEqual("core.csv.v1", stage.csv_source.parser_profile)
|
||||
self.assertNotIn("text", stage.metadata)
|
||||
self.assertEqual(2, self.writer.promote_stage.call_count)
|
||||
self.assertEqual(2, self.session.commit.call_count)
|
||||
|
||||
def test_json_stage_does_not_invent_csv_evidence(self) -> None:
|
||||
api_create_source_snapshot(
|
||||
TabularSnapshotCreateRequest(
|
||||
name="Upload", source_name="upload", rows=[{"value": "001"}]
|
||||
),
|
||||
session=self.session,
|
||||
principal=self.principal,
|
||||
)
|
||||
stage = self.writer.create_stage.call_args.kwargs["stage"]
|
||||
self.assertIsNone(stage.csv_source)
|
||||
self.assertEqual(({"value": "001"},), stage.rows)
|
||||
|
||||
def test_malformed_text_csv_is_422_before_any_durable_write(self) -> None:
|
||||
with self.assertRaises(HTTPException) as raised:
|
||||
api_create_source_snapshot(
|
||||
TabularSnapshotCreateRequest(
|
||||
name="Upload",
|
||||
source_name="upload",
|
||||
format="csv",
|
||||
csv_text="a,b\nonly-one\n",
|
||||
csv_value_mode="text",
|
||||
),
|
||||
session=self.session,
|
||||
principal=self.principal,
|
||||
)
|
||||
self.assertEqual(422, raised.exception.status_code)
|
||||
self.writer.create_stage.assert_not_called()
|
||||
self.writer.promote_stage.assert_not_called()
|
||||
self.session.commit.assert_not_called()
|
||||
|
||||
def test_invalid_unicode_is_rejected_by_request_schema_before_any_durable_write(
|
||||
self,
|
||||
) -> None:
|
||||
with self.assertRaises(ValidationError):
|
||||
api_create_source_snapshot(
|
||||
TabularSnapshotCreateRequest(
|
||||
name="Upload",
|
||||
source_name="upload",
|
||||
format="csv",
|
||||
csv_text="value\nprivate-\ud800\n",
|
||||
),
|
||||
session=self.session,
|
||||
principal=self.principal,
|
||||
)
|
||||
self.writer.create_stage.assert_not_called()
|
||||
self.writer.promote_stage.assert_not_called()
|
||||
self.session.commit.assert_not_called()
|
||||
@@ -0,0 +1,19 @@
|
||||
from govoplan_dataflow.backend.manifest import get_manifest
|
||||
|
||||
|
||||
def test_static_documentation_has_complete_german_reference_copy() -> None:
|
||||
for topic in get_manifest().documentation:
|
||||
german = topic.translations.get("de", {})
|
||||
assert all(german.get(field, "").strip() for field in ("title", "summary", "body")), topic.id
|
||||
|
||||
|
||||
def test_documentation_exposes_conditioned_workflow_and_reference() -> None:
|
||||
topics = {topic.id: topic for topic in get_manifest().documentation}
|
||||
workflow = topics["dataflow.module-boundary"]
|
||||
assert workflow.metadata.get("kind") == "workflow"
|
||||
assert any(condition.required_scopes for condition in workflow.conditions)
|
||||
assert workflow.structured_translations.get("de")
|
||||
|
||||
reference = topics["dataflow.reference.fields-and-consequences"]
|
||||
assert reference.metadata.get("kind") == "reference"
|
||||
assert reference.metadata.get("consequence_classes")
|
||||
@@ -0,0 +1,546 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import unittest
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.dsar import (
|
||||
DsarErasureActionRef,
|
||||
DsarProvider,
|
||||
DsarRecordRef,
|
||||
DsarSubjectRef,
|
||||
)
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_core.privacy.dsar_workflow import (
|
||||
create_data_subject_request,
|
||||
search_data_subject_request,
|
||||
)
|
||||
from govoplan_dataflow.backend.db.models import (
|
||||
DataflowPipeline,
|
||||
DataflowPipelineDeployment,
|
||||
DataflowPipelineRevision,
|
||||
DataflowReconciliationDecision,
|
||||
DataflowReconciliationDecisionSet,
|
||||
DataflowRun,
|
||||
DataflowTrigger,
|
||||
DataflowTriggerDelivery,
|
||||
)
|
||||
from govoplan_dataflow.backend.dsar_provider import (
|
||||
DATAFLOW_DSAR_CAPABILITY,
|
||||
DataflowDsarProvider,
|
||||
)
|
||||
from govoplan_dataflow.backend.manifest import manifest
|
||||
|
||||
|
||||
NOW = datetime(2026, 8, 21, 20, 0, tzinfo=UTC)
|
||||
SECRET = "private-dataflow-detail-do-not-export"
|
||||
|
||||
|
||||
class _Registry:
|
||||
def __init__(self, provider: DataflowDsarProvider, *, active: bool = True) -> None:
|
||||
self.provider = provider
|
||||
self.active = active
|
||||
|
||||
def capability_names(self):
|
||||
return (DATAFLOW_DSAR_CAPABILITY,)
|
||||
|
||||
def capability_owner(self, name):
|
||||
self._assert_capability(name)
|
||||
return "dataflow"
|
||||
|
||||
def tenant_entitlement_resolver(self):
|
||||
active = self.active
|
||||
|
||||
class _Resolver:
|
||||
@staticmethod
|
||||
def resolve(session, tenant_id):
|
||||
del session, tenant_id
|
||||
return type(
|
||||
"State",
|
||||
(),
|
||||
{"effective_modules": ("dataflow",) if active else ()},
|
||||
)()
|
||||
|
||||
return _Resolver()
|
||||
|
||||
def require_tenant_capability(self, name, session, **kwargs):
|
||||
del session, kwargs
|
||||
self._assert_capability(name)
|
||||
return self.provider
|
||||
|
||||
def manifests(self):
|
||||
return (type("Manifest", (), {"id": "dataflow"})(),)
|
||||
|
||||
@staticmethod
|
||||
def _assert_capability(name: str) -> None:
|
||||
if name != DATAFLOW_DSAR_CAPABILITY:
|
||||
raise KeyError(name)
|
||||
|
||||
|
||||
class DataflowDsarProviderTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(self.engine)
|
||||
self.session = Session(self.engine)
|
||||
self.provider = DataflowDsarProvider()
|
||||
self.assertIsInstance(self.provider, DsarProvider)
|
||||
self._seed()
|
||||
self.session.commit()
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.session.close()
|
||||
self.engine.dispose()
|
||||
|
||||
def _seed(self) -> None:
|
||||
pipeline = DataflowPipeline(
|
||||
id="pipeline-1",
|
||||
tenant_id="tenant-1",
|
||||
scope_type="tenant",
|
||||
definition_kind="flow",
|
||||
name=SECRET,
|
||||
description=SECRET,
|
||||
status="active",
|
||||
current_revision=1,
|
||||
created_by="account-1",
|
||||
updated_by="account-1",
|
||||
metadata_={"secret": SECRET},
|
||||
derivation_provenance={"secret": SECRET},
|
||||
)
|
||||
other = DataflowPipeline(
|
||||
id="pipeline-other",
|
||||
tenant_id="tenant-2",
|
||||
scope_type="tenant",
|
||||
definition_kind="flow",
|
||||
name=SECRET,
|
||||
status="active",
|
||||
current_revision=1,
|
||||
created_by="account-1",
|
||||
updated_by="account-1",
|
||||
)
|
||||
self.session.add_all((pipeline, other))
|
||||
self.session.flush()
|
||||
revision = DataflowPipelineRevision(
|
||||
id="revision-1",
|
||||
tenant_id="tenant-1",
|
||||
pipeline_id=pipeline.id,
|
||||
revision=1,
|
||||
schema_version=1,
|
||||
graph={"secret": SECRET},
|
||||
sql_text=SECRET,
|
||||
editor_mode="graph",
|
||||
content_hash="a" * 64,
|
||||
created_by="account-1",
|
||||
)
|
||||
self.session.add(revision)
|
||||
self.session.flush()
|
||||
decision_set = DataflowReconciliationDecisionSet(
|
||||
id="decision-set-1",
|
||||
tenant_id="tenant-1",
|
||||
pipeline_id=pipeline.id,
|
||||
name=SECRET,
|
||||
node_id="reconcile",
|
||||
created_by="account-1",
|
||||
updated_by="account-1",
|
||||
)
|
||||
self.session.add(decision_set)
|
||||
self.session.flush()
|
||||
run = DataflowRun(
|
||||
id="run-1",
|
||||
tenant_id="tenant-1",
|
||||
pipeline_id=pipeline.id,
|
||||
pipeline_revision_id=revision.id,
|
||||
run_type="execute",
|
||||
status="succeeded",
|
||||
execution_backend="python",
|
||||
environment="production",
|
||||
executor_version="dataflow-v1",
|
||||
definition_hash="b" * 64,
|
||||
idempotency_key=SECRET,
|
||||
request_hash="c" * 64,
|
||||
request_={"secret": SECRET},
|
||||
invocation_kind="manual",
|
||||
correlation_id=SECRET,
|
||||
causation_id=SECRET,
|
||||
source_fingerprints=[{"secret": SECRET}],
|
||||
result_schema=[{"secret": SECRET}],
|
||||
diagnostics=[{"secret": SECRET}],
|
||||
input_row_count=1,
|
||||
output_row_count=1,
|
||||
output_publication_ref=SECRET,
|
||||
output_datasource_ref=SECRET,
|
||||
output_materialization_ref=SECRET,
|
||||
attempts=1,
|
||||
progress_percent=100,
|
||||
progress_phase="complete",
|
||||
retention_until=NOW + timedelta(days=30),
|
||||
authorization_={"submitted_principal": {"secret": SECRET}},
|
||||
resource_budget={"secret": SECRET},
|
||||
started_at=NOW,
|
||||
finished_at=NOW,
|
||||
error=SECRET,
|
||||
created_by="account-1",
|
||||
)
|
||||
trigger = DataflowTrigger(
|
||||
id="trigger-1",
|
||||
tenant_id="tenant-1",
|
||||
pipeline_id=pipeline.id,
|
||||
pipeline_revision_id=revision.id,
|
||||
name=SECRET,
|
||||
kind="event",
|
||||
status="active",
|
||||
revision=1,
|
||||
config_={"secret": SECRET},
|
||||
publication_={"secret": SECRET},
|
||||
row_limit=100,
|
||||
catch_up_policy="coalesce",
|
||||
max_concurrent_runs=1,
|
||||
next_fire_at=NOW + timedelta(hours=1),
|
||||
last_error=SECRET,
|
||||
authorization_account_id="account-1",
|
||||
authorization_membership_id="membership-1",
|
||||
authorization_ref=SECRET,
|
||||
grant_scopes=[SECRET],
|
||||
created_by="account-1",
|
||||
updated_by="account-1",
|
||||
)
|
||||
self.session.add_all(
|
||||
(
|
||||
DataflowReconciliationDecision(
|
||||
id="decision-1",
|
||||
tenant_id="tenant-1",
|
||||
decision_set_id=decision_set.id,
|
||||
revision=1,
|
||||
key_hash="d" * 64,
|
||||
input_hash="e" * 64,
|
||||
action="correct",
|
||||
reason=SECRET,
|
||||
correction={"secret": SECRET},
|
||||
actor_ref="account-1",
|
||||
decided_at=NOW,
|
||||
),
|
||||
run,
|
||||
DataflowPipelineDeployment(
|
||||
id="deployment-1",
|
||||
tenant_id="tenant-1",
|
||||
pipeline_id=pipeline.id,
|
||||
pipeline_revision_id=revision.id,
|
||||
environment="production",
|
||||
source_environment="staging",
|
||||
status="active",
|
||||
provenance={"secret": SECRET},
|
||||
promoted_by="account-1",
|
||||
),
|
||||
trigger,
|
||||
)
|
||||
)
|
||||
self.session.flush()
|
||||
self.session.add(
|
||||
DataflowTriggerDelivery(
|
||||
id="delivery-1",
|
||||
tenant_id="tenant-1",
|
||||
trigger_id=trigger.id,
|
||||
pipeline_id=pipeline.id,
|
||||
pipeline_revision_id=revision.id,
|
||||
source_key=SECRET,
|
||||
invocation_kind="event",
|
||||
status="succeeded",
|
||||
scheduled_for=NOW,
|
||||
event_={"secret": SECRET},
|
||||
run_id=run.id,
|
||||
attempts=1,
|
||||
authorization_provenance={"secret": SECRET},
|
||||
error=SECRET,
|
||||
finished_at=NOW,
|
||||
)
|
||||
)
|
||||
|
||||
def test_canonical_selector_is_minimized_and_marks_automation_authority(
|
||||
self,
|
||||
) -> None:
|
||||
records = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=DsarSubjectRef(
|
||||
account_id="account-1",
|
||||
membership_id="membership-1",
|
||||
),
|
||||
)
|
||||
|
||||
self.assertEqual(7, len(records))
|
||||
trigger = next(
|
||||
item for item in records if item.resource_type == "dataflow_trigger"
|
||||
)
|
||||
self.assertEqual("dataflow_automation_authority", trigger.category)
|
||||
exported = json.dumps([record.to_dict() for record in records])
|
||||
self.assertNotIn(SECRET, exported)
|
||||
self.assertNotIn("account-1", exported)
|
||||
self.assertNotIn("membership-1", exported)
|
||||
self.assertNotIn("pipeline-other", exported)
|
||||
|
||||
def test_exact_pipeline_package_is_review_only_and_minimized(self) -> None:
|
||||
records = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=DsarSubjectRef(
|
||||
account_id="account-1",
|
||||
external_references={"dataflow.pipeline": "pipeline:pipeline-1"},
|
||||
),
|
||||
)
|
||||
|
||||
self.assertEqual(8, len(records))
|
||||
exported = json.dumps([record.to_dict() for record in records])
|
||||
self.assertNotIn(SECRET, exported)
|
||||
actions = self.provider.plan_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=DsarSubjectRef(
|
||||
external_references={"dataflow.pipeline": "pipeline-1"}
|
||||
),
|
||||
records=records,
|
||||
)
|
||||
self.assertEqual({"manual_review"}, {action.kind for action in actions})
|
||||
|
||||
def test_every_exact_reference_and_fail_closed_correlation(self) -> None:
|
||||
references = {
|
||||
"dataflow.pipeline_revision": "revision-1",
|
||||
"dataflow.decision_set": "decision-set-1",
|
||||
"dataflow.decision": "decision-1",
|
||||
"dataflow.run": "dataflow-run:run-1",
|
||||
"dataflow.deployment": "deployment-1",
|
||||
"dataflow.trigger": "trigger-1",
|
||||
"dataflow.trigger_delivery": "delivery-1",
|
||||
}
|
||||
for key, value in references.items():
|
||||
with self.subTest(key=key):
|
||||
records = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=DsarSubjectRef(external_references={key: value}),
|
||||
)
|
||||
self.assertEqual(1, len(records))
|
||||
|
||||
mismatch = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=DsarSubjectRef(
|
||||
account_id="account-2",
|
||||
external_references={"dataflow.run": "run-1"},
|
||||
),
|
||||
)
|
||||
wrong_tenant = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-2",
|
||||
subject=DsarSubjectRef(external_references={"dataflow.run": "run-1"}),
|
||||
)
|
||||
conflict = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=DsarSubjectRef(
|
||||
external_references={
|
||||
"dataflow.pipeline_revision": "revision-1",
|
||||
"dataflow.revision": "different",
|
||||
}
|
||||
),
|
||||
)
|
||||
self.assertEqual((), mismatch)
|
||||
self.assertEqual((), wrong_tenant)
|
||||
self.assertEqual((), conflict)
|
||||
|
||||
def test_terminal_run_and_delivery_minimization_is_idempotent(self) -> None:
|
||||
subject = DsarSubjectRef(
|
||||
external_references={
|
||||
"dataflow.run": "run-1",
|
||||
"dataflow.delivery": "delivery-1",
|
||||
}
|
||||
)
|
||||
records = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=subject,
|
||||
)
|
||||
actions = self.provider.plan_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=subject,
|
||||
records=records,
|
||||
)
|
||||
self.assertEqual({"anonymize"}, {action.kind for action in actions})
|
||||
first = self.provider.execute_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=subject,
|
||||
actions=actions,
|
||||
request_id="dsar-1",
|
||||
)
|
||||
second = self.provider.execute_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=subject,
|
||||
actions=actions,
|
||||
request_id="dsar-1-retry",
|
||||
)
|
||||
self.assertTrue(all(result.status == "executed" for result in first))
|
||||
self.assertTrue(all(result.status == "unchanged" for result in second))
|
||||
run = self.session.get(DataflowRun, "run-1")
|
||||
delivery = self.session.get(DataflowTriggerDelivery, "delivery-1")
|
||||
self.assertEqual({}, run.request_)
|
||||
self.assertEqual([], run.diagnostics)
|
||||
self.assertIsNotNone(run.purged_at)
|
||||
self.assertIsNone(delivery.event_)
|
||||
self.assertEqual({}, delivery.authorization_provenance)
|
||||
self.assertEqual(SECRET, delivery.source_key)
|
||||
|
||||
def test_automation_authority_is_revoked_with_retry_support(self) -> None:
|
||||
subject = DsarSubjectRef(
|
||||
account_id="account-1",
|
||||
membership_id="membership-1",
|
||||
)
|
||||
records = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=subject,
|
||||
)
|
||||
actions = self.provider.plan_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=subject,
|
||||
records=records,
|
||||
)
|
||||
trigger_action = next(
|
||||
action for action in actions if action.resource_type == "dataflow_trigger"
|
||||
)
|
||||
self.assertEqual("revoke", trigger_action.kind)
|
||||
self.assertTrue(trigger_action.executable)
|
||||
first = self.provider.execute_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=subject,
|
||||
actions=(trigger_action,),
|
||||
request_id="dsar-2",
|
||||
)
|
||||
second = self.provider.execute_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=subject,
|
||||
actions=(trigger_action,),
|
||||
request_id="dsar-2-retry",
|
||||
)
|
||||
self.assertEqual("executed", first[0].status)
|
||||
self.assertEqual("unchanged", second[0].status)
|
||||
trigger = self.session.get(DataflowTrigger, "trigger-1")
|
||||
self.assertEqual("disabled", trigger.status)
|
||||
self.assertEqual("redacted", trigger.authorization_account_id)
|
||||
self.assertEqual("redacted", trigger.authorization_membership_id)
|
||||
self.assertEqual([], trigger.grant_scopes)
|
||||
self.assertEqual({}, trigger.config_)
|
||||
|
||||
def test_foreign_records_and_actions_are_rejected(self) -> None:
|
||||
subject = DsarSubjectRef(account_id="account-1")
|
||||
with self.assertRaisesRegex(ValueError, "foreign provider record"):
|
||||
self.provider.plan_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=subject,
|
||||
records=(
|
||||
DsarRecordRef(
|
||||
provider_id="cases",
|
||||
module_id="cases",
|
||||
resource_type="case",
|
||||
resource_id="case-1",
|
||||
category="case",
|
||||
title="Case",
|
||||
),
|
||||
),
|
||||
)
|
||||
with self.assertRaisesRegex(ValueError, "foreign provider action"):
|
||||
self.provider.execute_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=subject,
|
||||
actions=(
|
||||
DsarErasureActionRef(
|
||||
action_id="cases:delete:case:case-1",
|
||||
provider_id="cases",
|
||||
module_id="cases",
|
||||
kind="delete",
|
||||
resource_type="case",
|
||||
resource_id="case-1",
|
||||
title="Delete case",
|
||||
rationale="Foreign",
|
||||
executable=True,
|
||||
),
|
||||
),
|
||||
request_id="dsar-3",
|
||||
)
|
||||
|
||||
def test_core_workflow_reports_active_and_inactive_provider(self) -> None:
|
||||
row = create_data_subject_request(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
reference="DSAR-DATAFLOW-1",
|
||||
request_kind="access_and_erasure",
|
||||
subject=DsarSubjectRef(account_id="account-1"),
|
||||
purpose="Respond to a verified request.",
|
||||
legal_basis="Article 15 and 17 GDPR",
|
||||
due_at=None,
|
||||
requested_by_account_id="privacy-officer",
|
||||
)
|
||||
self.session.commit()
|
||||
search_data_subject_request(
|
||||
self.session,
|
||||
registry=_Registry(self.provider),
|
||||
row=row,
|
||||
expected_revision=1,
|
||||
)
|
||||
self.assertEqual(
|
||||
[DATAFLOW_DSAR_CAPABILITY],
|
||||
row.coverage["provider_capabilities"],
|
||||
)
|
||||
self.assertEqual(7, row.search_result["record_count"])
|
||||
|
||||
inactive = create_data_subject_request(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
reference="DSAR-DATAFLOW-2",
|
||||
request_kind="access",
|
||||
subject=DsarSubjectRef(account_id="account-1"),
|
||||
purpose="Respond to a verified request.",
|
||||
legal_basis="Article 15 GDPR",
|
||||
due_at=None,
|
||||
requested_by_account_id="privacy-officer",
|
||||
)
|
||||
self.session.commit()
|
||||
search_data_subject_request(
|
||||
self.session,
|
||||
registry=_Registry(self.provider, active=False),
|
||||
row=inactive,
|
||||
expected_revision=1,
|
||||
)
|
||||
self.assertEqual([], inactive.coverage["provider_capabilities"])
|
||||
self.assertEqual(
|
||||
[DATAFLOW_DSAR_CAPABILITY],
|
||||
inactive.coverage["inactive_provider_capabilities"],
|
||||
)
|
||||
self.assertEqual(0, inactive.search_result["record_count"])
|
||||
|
||||
def test_manifest_registers_and_documents_capability(self) -> None:
|
||||
self.assertIn(DATAFLOW_DSAR_CAPABILITY, manifest.capability_factories)
|
||||
self.assertIn(DATAFLOW_DSAR_CAPABILITY, manifest.capability_documentation)
|
||||
self.assertIn(
|
||||
DATAFLOW_DSAR_CAPABILITY,
|
||||
{item.name for item in manifest.provides_interfaces},
|
||||
)
|
||||
self.assertTrue(
|
||||
any(
|
||||
topic.id == "dataflow.data-subject-requests"
|
||||
and {"admin", "user"}.issubset(topic.documentation_types)
|
||||
for topic in manifest.documentation
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -22,6 +22,7 @@ class DataflowInterfaceDocumentationContractTests(unittest.TestCase):
|
||||
"dataflow.sql",
|
||||
"dataflow.inspector",
|
||||
"dataflow.results",
|
||||
"dataflow.decisions",
|
||||
"dataflow.triggers",
|
||||
"dataflow.runs",
|
||||
"dataflow.widget.pipelines",
|
||||
@@ -38,6 +39,7 @@ class DataflowInterfaceDocumentationContractTests(unittest.TestCase):
|
||||
"dataflow.runs",
|
||||
):
|
||||
self.assertEqual("dataflow.page", surfaces[surface_id].parent_id)
|
||||
self.assertEqual("dataflow.results", surfaces["dataflow.decisions"].parent_id)
|
||||
|
||||
def test_help_and_consequence_metadata_remain_published(self) -> None:
|
||||
topics = {topic.id: topic for topic in get_manifest().documentation}
|
||||
@@ -48,8 +50,10 @@ class DataflowInterfaceDocumentationContractTests(unittest.TestCase):
|
||||
|
||||
self.assertIn("dataflow.state.read-only", boundary.metadata["help_contexts"])
|
||||
self.assertIn("dataflow.field.expression", nodes.metadata["help_contexts"])
|
||||
self.assertIn("dataflow.action.review-decisions", nodes.metadata["help_contexts"])
|
||||
self.assertIn("save_revision", fields.metadata["consequence_classes"])
|
||||
self.assertIn("delete_pipeline", fields.metadata["consequence_classes"])
|
||||
self.assertIn("record_decision", fields.metadata["consequence_classes"])
|
||||
self.assertIn("publish_output", execution.metadata["consequence_classes"])
|
||||
self.assertIn("promote_revision", execution.metadata["consequence_classes"])
|
||||
|
||||
@@ -63,12 +67,13 @@ class DataflowInterfaceDocumentationContractTests(unittest.TestCase):
|
||||
|
||||
for component in (
|
||||
"ActionBlockerHint",
|
||||
"DocumentationHelpLink",
|
||||
"titleHelp={<DocumentationHelpLink reference={DATAFLOW_DOCUMENTATION} />}",
|
||||
"useUnsavedDraftGuard",
|
||||
"ConfirmDialog",
|
||||
):
|
||||
self.assertIn(component, page)
|
||||
self.assertIn("DATAFLOW_NODE_DOCUMENTATION", inspector)
|
||||
self.assertNotIn("helpAction=", page)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -44,6 +44,7 @@ class DataflowManifestTests(unittest.TestCase):
|
||||
"dataflow.run_worker",
|
||||
"dataflow.dataset_output",
|
||||
"dataflow.trigger_dispatcher",
|
||||
"privacy.dsar.dataflow",
|
||||
},
|
||||
{item.name for item in manifest.provides_interfaces},
|
||||
)
|
||||
|
||||
@@ -24,7 +24,7 @@ class DataflowMigrationTests(unittest.TestCase):
|
||||
try:
|
||||
with engine.connect() as connection:
|
||||
self.assertIn(
|
||||
"f6c2a9d4e7b1",
|
||||
"a3d7f1c5e9b2",
|
||||
set(MigrationContext.configure(connection).get_current_heads()),
|
||||
)
|
||||
self.assertEqual(
|
||||
@@ -32,6 +32,8 @@ class DataflowMigrationTests(unittest.TestCase):
|
||||
"dataflow_pipelines",
|
||||
"dataflow_pipeline_revisions",
|
||||
"dataflow_pipeline_deployments",
|
||||
"dataflow_reconciliation_decision_sets",
|
||||
"dataflow_reconciliation_decisions",
|
||||
"dataflow_runs",
|
||||
"dataflow_triggers",
|
||||
"dataflow_trigger_deliveries",
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
|
||||
from govoplan_dataflow.backend.executor import (
|
||||
MAX_RESULT_BYTES,
|
||||
PipelineExecutionError,
|
||||
execute_preview,
|
||||
)
|
||||
from govoplan_dataflow.backend.expressions import ExpressionError, evaluate_expression
|
||||
from govoplan_dataflow.backend.manifest import get_manifest
|
||||
from govoplan_dataflow.backend.schemas import (
|
||||
GraphEdge,
|
||||
GraphNode,
|
||||
GraphPosition,
|
||||
PipelineGraph,
|
||||
)
|
||||
|
||||
|
||||
class _UnreadableFill:
|
||||
def __str__(self) -> str:
|
||||
raise AssertionError(
|
||||
"Oversized padding must be rejected before reading or multiplying its fill."
|
||||
)
|
||||
|
||||
|
||||
def _expression_graph(expression: str, *, result_type: str = "string") -> PipelineGraph:
|
||||
return PipelineGraph(
|
||||
nodes=[
|
||||
GraphNode(
|
||||
id="source",
|
||||
type="source.inline",
|
||||
label="Source",
|
||||
position=GraphPosition(x=0, y=0),
|
||||
config={"source_name": "fixture", "rows": [{"value": "x"}]},
|
||||
),
|
||||
GraphNode(
|
||||
id="padding",
|
||||
type="expression",
|
||||
label="Padding",
|
||||
position=GraphPosition(x=200, y=0),
|
||||
config={
|
||||
"target_column": "padded",
|
||||
"expression": expression,
|
||||
"result_type": result_type,
|
||||
},
|
||||
),
|
||||
GraphNode(
|
||||
id="output",
|
||||
type="output",
|
||||
label="Output",
|
||||
position=GraphPosition(x=400, y=0),
|
||||
config={},
|
||||
),
|
||||
],
|
||||
edges=[
|
||||
GraphEdge(id="source-padding", source="source", target="padding"),
|
||||
GraphEdge(id="padding-output", source="padding", target="output"),
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
class PaddingBudgetTests(unittest.TestCase):
|
||||
def test_oversized_padding_is_rejected_before_fill_evaluation_or_allocation(
|
||||
self,
|
||||
) -> None:
|
||||
for operation in ("lpad", "rpad"):
|
||||
for length in (MAX_RESULT_BYTES + 1, 10**100):
|
||||
with self.subTest(operation=operation, length=length):
|
||||
with self.assertRaisesRegex(
|
||||
ExpressionError, "Padding length.*1,000,000"
|
||||
):
|
||||
evaluate_expression(
|
||||
f"{operation}('x', {length}, fill)",
|
||||
{"fill": _UnreadableFill()},
|
||||
)
|
||||
|
||||
def test_budget_cannot_be_bypassed_by_hiding_large_padding_in_a_small_scalar(
|
||||
self,
|
||||
) -> None:
|
||||
for wrapper in ("length({})", "substring({}, 1, 1)"):
|
||||
with self.subTest(wrapper=wrapper):
|
||||
expression = wrapper.format(f"lpad('x', {MAX_RESULT_BYTES + 1}, fill)")
|
||||
with self.assertRaises(ExpressionError):
|
||||
evaluate_expression(expression, {"fill": _UnreadableFill()})
|
||||
|
||||
def test_existing_boundary_and_ordinary_padding_are_preserved(self) -> None:
|
||||
self.assertEqual(1_000_000, MAX_RESULT_BYTES)
|
||||
for operation in ("lpad", "rpad"):
|
||||
with self.subTest(operation=operation):
|
||||
value = evaluate_expression(
|
||||
f"{operation}('x', {MAX_RESULT_BYTES}, '0')", {}
|
||||
)
|
||||
self.assertEqual(MAX_RESULT_BYTES, len(value))
|
||||
self.assertEqual(1, value.count("x"))
|
||||
self.assertEqual(
|
||||
"abc", evaluate_expression(f"{operation}('abcdef', 3, '')", {})
|
||||
)
|
||||
self.assertEqual(
|
||||
"", evaluate_expression(f"{operation}('abcdef', 0, '')", {})
|
||||
)
|
||||
self.assertEqual(
|
||||
"abc", evaluate_expression(f"{operation}('abc', 3, '')", {})
|
||||
)
|
||||
|
||||
def test_null_negative_and_empty_fill_semantics_are_unchanged(self) -> None:
|
||||
for operation in ("lpad", "rpad"):
|
||||
with self.subTest(operation=operation):
|
||||
self.assertIsNone(
|
||||
evaluate_expression(f"{operation}(NULL, {10**100}, '')", {})
|
||||
)
|
||||
self.assertIsNone(evaluate_expression(f"{operation}(NULL, -1, '')", {}))
|
||||
with self.assertRaisesRegex(ValueError, "cannot be negative"):
|
||||
evaluate_expression(f"{operation}('x', -1, '0')", {})
|
||||
with self.assertRaisesRegex(ValueError, "fill text cannot be empty"):
|
||||
evaluate_expression(f"{operation}('x', 2, '')", {})
|
||||
|
||||
def test_multibyte_fill_and_truncation_preserve_character_semantics(self) -> None:
|
||||
self.assertEqual("ö🙂öÄ", evaluate_expression("lpad('Ä', 4, 'ö🙂')", {}))
|
||||
self.assertEqual("Äö🙂ö", evaluate_expression("rpad('Ä', 4, 'ö🙂')", {}))
|
||||
self.assertEqual("🙂ä", evaluate_expression("lpad('🙂ä中', 2, '0')", {}))
|
||||
result = execute_preview(
|
||||
_expression_graph("rpad(value, 4, 'ö🙂')"), row_limit=10
|
||||
)
|
||||
self.assertEqual("xö🙂ö", result.rows[0]["padded"])
|
||||
|
||||
def test_preview_reports_padding_guard_at_owning_node_and_retains_final_byte_limit(
|
||||
self,
|
||||
) -> None:
|
||||
with self.assertRaisesRegex(PipelineExecutionError, "Padding length") as raised:
|
||||
execute_preview(
|
||||
_expression_graph(
|
||||
f"length(lpad(value, {MAX_RESULT_BYTES + 1}, '0'))",
|
||||
result_type="integer",
|
||||
),
|
||||
row_limit=10,
|
||||
)
|
||||
self.assertEqual("padding", raised.exception.node_id)
|
||||
# Non-ASCII characters need several serialized bytes. The preallocation
|
||||
# character bound supplements, and never replaces, the node byte bound.
|
||||
with self.assertRaisesRegex(
|
||||
PipelineExecutionError, "one-megabyte result limit"
|
||||
) as raised:
|
||||
execute_preview(_expression_graph("rpad(value, 200000, 'ö')"), row_limit=10)
|
||||
self.assertEqual("padding", raised.exception.node_id)
|
||||
|
||||
def test_user_and_operator_documentation_explains_intermediate_padding_limit_in_both_languages(
|
||||
self,
|
||||
) -> None:
|
||||
topic = next(
|
||||
topic
|
||||
for topic in get_manifest().documentation
|
||||
if topic.id == "dataflow.reference.nodes-and-expressions"
|
||||
)
|
||||
self.assertIn("user", topic.documentation_types)
|
||||
self.assertIn("admin", topic.documentation_types)
|
||||
for text in (topic.body, topic.translations["de"]["body"]):
|
||||
self.assertIn("LPAD", text)
|
||||
self.assertIn("RPAD", text)
|
||||
self.assertIn("LENGTH", text)
|
||||
self.assertIn("SUBSTRING", text)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,247 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.core.access import PrincipalRef
|
||||
from govoplan_core.core.concurrency import RevisionConflictError
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_dataflow.backend.db.models import (
|
||||
DataflowPipeline,
|
||||
DataflowReconciliationDecision,
|
||||
DataflowReconciliationDecisionSet,
|
||||
)
|
||||
from govoplan_dataflow.backend.executor import PipelineExecutionError
|
||||
from govoplan_dataflow.backend.reconciliation_decisions import (
|
||||
create_decision_set,
|
||||
current_decision_rows,
|
||||
current_decisions,
|
||||
decision_set_fingerprint,
|
||||
decision_set_payload,
|
||||
decision_set_source_payload,
|
||||
record_decision,
|
||||
)
|
||||
from govoplan_dataflow.backend.schemas import GraphNode, GraphPosition
|
||||
from govoplan_dataflow.backend.service import _datasource_source_resolver
|
||||
|
||||
|
||||
TABLES = (
|
||||
DataflowPipeline.__table__,
|
||||
DataflowReconciliationDecisionSet.__table__,
|
||||
DataflowReconciliationDecision.__table__,
|
||||
)
|
||||
|
||||
|
||||
class ReconciliationDecisionTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.engine = create_engine("sqlite:///:memory:")
|
||||
Base.metadata.create_all(self.engine, tables=TABLES)
|
||||
self.session = Session(self.engine)
|
||||
self.pipeline = DataflowPipeline(
|
||||
tenant_id="tenant-1",
|
||||
scope_type="tenant",
|
||||
scope_id="tenant-1",
|
||||
name="Monthly reconciliation",
|
||||
status="active",
|
||||
created_by="account-1",
|
||||
updated_by="account-1",
|
||||
)
|
||||
self.session.add(self.pipeline)
|
||||
self.session.flush()
|
||||
self.principal = ApiPrincipal(
|
||||
principal=PrincipalRef(
|
||||
account_id="account-1",
|
||||
membership_id="membership-1",
|
||||
tenant_id="tenant-1",
|
||||
scopes=frozenset(
|
||||
{
|
||||
"dataflow:pipeline:read",
|
||||
"dataflow:pipeline:write",
|
||||
"dataflow:pipeline:run",
|
||||
}
|
||||
),
|
||||
),
|
||||
account=SimpleNamespace(id="account-1"),
|
||||
user=SimpleNamespace(id="membership-1"),
|
||||
)
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.session.close()
|
||||
self.engine.dispose()
|
||||
|
||||
def test_decisions_are_immutable_current_rows_and_occ_protected(self) -> None:
|
||||
decision_set = create_decision_set(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
pipeline_id=self.pipeline.id,
|
||||
name="July review",
|
||||
node_id="reconcile",
|
||||
actor_ref="account:account-1",
|
||||
)
|
||||
first = record_decision(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
decision_set_id=decision_set.id,
|
||||
expected_revision=1,
|
||||
key_hash="a" * 64,
|
||||
input_hash="b" * 64,
|
||||
action="accept",
|
||||
reason="Checked against the source record.",
|
||||
correction=None,
|
||||
actor_ref="account:account-1",
|
||||
)
|
||||
first_fingerprint = decision_set_fingerprint(first)
|
||||
second = record_decision(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
decision_set_id=decision_set.id,
|
||||
expected_revision=2,
|
||||
key_hash="a" * 64,
|
||||
input_hash="c" * 64,
|
||||
action="correct",
|
||||
reason="The monthly source changed after review.",
|
||||
correction={"amount": 25},
|
||||
actor_ref="account:account-1",
|
||||
)
|
||||
|
||||
self.assertEqual(2, len(second.decisions))
|
||||
self.assertEqual(1, len(current_decisions(second)))
|
||||
self.assertEqual("c" * 64, current_decisions(second)[0].input_hash)
|
||||
self.assertNotEqual(first_fingerprint, decision_set_fingerprint(second))
|
||||
payload = decision_set_payload(second)
|
||||
self.assertEqual(2, len(payload["history"]))
|
||||
self.assertEqual(1, len(payload["current_decisions"]))
|
||||
with self.assertRaises(RevisionConflictError):
|
||||
record_decision(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
decision_set_id=decision_set.id,
|
||||
expected_revision=2,
|
||||
key_hash="d" * 64,
|
||||
input_hash="e" * 64,
|
||||
action="defer",
|
||||
reason="Needs another source.",
|
||||
correction=None,
|
||||
actor_ref="account:account-1",
|
||||
)
|
||||
|
||||
def test_decision_set_is_a_fingerprinted_reference_source(self) -> None:
|
||||
decision_set = create_decision_set(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
pipeline_id=self.pipeline.id,
|
||||
name="July review",
|
||||
node_id="reconcile",
|
||||
actor_ref="account:account-1",
|
||||
)
|
||||
record_decision(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
decision_set_id=decision_set.id,
|
||||
expected_revision=1,
|
||||
key_hash="a" * 64,
|
||||
input_hash="b" * 64,
|
||||
action="reject",
|
||||
reason="The observed record belongs to another case.",
|
||||
correction=None,
|
||||
actor_ref="account:account-1",
|
||||
)
|
||||
source = decision_set_source_payload(decision_set)
|
||||
resolver = _datasource_source_resolver(
|
||||
session=self.session,
|
||||
principal=self.principal,
|
||||
registry=None,
|
||||
)
|
||||
node = GraphNode(
|
||||
id="decisions",
|
||||
type="source.reference",
|
||||
label="Review decisions",
|
||||
position=GraphPosition(x=0, y=0),
|
||||
config={
|
||||
"source_ref": source["ref"],
|
||||
"expected_fingerprint": source["fingerprint"],
|
||||
},
|
||||
)
|
||||
|
||||
resolved = resolver(node, 100)
|
||||
|
||||
self.assertEqual("dataflow.reconciliation_decisions", resolved.provider)
|
||||
self.assertEqual(1, resolved.total_rows)
|
||||
self.assertEqual("reject", resolved.rows[0]["action"])
|
||||
stale = node.model_copy(deep=True)
|
||||
stale.config["expected_fingerprint"] = "sha256:" + "0" * 64
|
||||
with self.assertRaisesRegex(PipelineExecutionError, "changed"):
|
||||
resolver(stale, 100)
|
||||
|
||||
def test_tenant_can_keep_decisions_for_visible_system_pipeline(self) -> None:
|
||||
system_pipeline = DataflowPipeline(
|
||||
tenant_id=None,
|
||||
scope_type="system",
|
||||
scope_id=None,
|
||||
name="Governed monthly reconciliation",
|
||||
status="active",
|
||||
created_by="system-admin",
|
||||
updated_by="system-admin",
|
||||
)
|
||||
self.session.add(system_pipeline)
|
||||
self.session.flush()
|
||||
|
||||
decision_set = create_decision_set(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
pipeline_id=system_pipeline.id,
|
||||
name="Tenant July review",
|
||||
node_id=None,
|
||||
actor_ref="account:account-1",
|
||||
)
|
||||
|
||||
self.assertEqual("tenant-1", decision_set.tenant_id)
|
||||
self.assertEqual(system_pipeline.id, decision_set.pipeline_id)
|
||||
|
||||
def test_summary_and_current_projection_do_not_require_full_history(self) -> None:
|
||||
decision_set = create_decision_set(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
pipeline_id=self.pipeline.id,
|
||||
name="Bounded review",
|
||||
node_id="reconcile",
|
||||
actor_ref="account:account-1",
|
||||
)
|
||||
for expected_revision, key_hash, input_hash in (
|
||||
(1, "a" * 64, "b" * 64),
|
||||
(2, "c" * 64, "d" * 64),
|
||||
(3, "a" * 64, "e" * 64),
|
||||
):
|
||||
record_decision(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
decision_set_id=decision_set.id,
|
||||
expected_revision=expected_revision,
|
||||
key_hash=key_hash,
|
||||
input_hash=input_hash,
|
||||
action="accept",
|
||||
reason="Reviewed against the current input.",
|
||||
correction=None,
|
||||
actor_ref="account:account-1",
|
||||
)
|
||||
|
||||
summary = decision_set_payload(decision_set, include_decisions=False)
|
||||
rows, total = current_decision_rows(
|
||||
self.session,
|
||||
decision_set_id=decision_set.id,
|
||||
limit=1,
|
||||
)
|
||||
|
||||
self.assertFalse(summary["decisions_included"])
|
||||
self.assertEqual([], summary["current_decisions"])
|
||||
self.assertEqual([], summary["history"])
|
||||
self.assertEqual(2, total)
|
||||
self.assertEqual(1, len(rows))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,117 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import date, datetime, timezone
|
||||
from decimal import Decimal
|
||||
import time
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from govoplan_core.security.bounded_process import ProcessBudgetError
|
||||
from govoplan_dataflow.backend.backends import (
|
||||
BackendExecutionError, BackendSource, ExecutionBudget, execute_typed_graph,
|
||||
)
|
||||
from govoplan_dataflow.backend.backends.reference import ReferenceExecutionBackend
|
||||
from govoplan_dataflow.backend.batches import TypedBatch
|
||||
from govoplan_dataflow.backend.executor import PipelineExecutionError, ResolvedSource
|
||||
from govoplan_dataflow.backend.manifest import get_manifest
|
||||
from govoplan_dataflow.backend.schemas import GraphEdge, GraphNode, GraphPosition, PipelineGraph
|
||||
from govoplan_dataflow.backend.service import _execute_pipeline_preview, _typed_backend_sources
|
||||
|
||||
|
||||
def graph_for(expression: str = "value", rows: list | None = None) -> PipelineGraph:
|
||||
nodes = [
|
||||
GraphNode(id="source", type="source.inline", label="Source", position=GraphPosition(x=0, y=0),
|
||||
config={"source_name": "records", "rows": rows or [{"value": "normal"}]}),
|
||||
GraphNode(id="expression", type="expression", label="Expression", position=GraphPosition(x=100, y=0),
|
||||
config={"target_column": "result", "expression": expression, "result_type": "unknown"}),
|
||||
GraphNode(id="output", type="output", label="Output", position=GraphPosition(x=200, y=0), config={}),
|
||||
]
|
||||
return PipelineGraph(nodes=nodes, edges=[
|
||||
GraphEdge(id="first", source="source", target="expression"),
|
||||
GraphEdge(id="second", source="expression", target="output"),
|
||||
])
|
||||
|
||||
|
||||
class ReferenceProcessTests(unittest.TestCase):
|
||||
def test_non_finite_deadlines_are_rejected_before_registry_wait(self) -> None:
|
||||
for value in (float("nan"), float("inf")):
|
||||
with self.subTest(value=value), self.assertRaisesRegex(ValueError, "finite"):
|
||||
ExecutionBudget(max_wall_seconds=value)
|
||||
|
||||
def test_real_child_not_parent_helper_and_typed_result_preserved(self) -> None:
|
||||
row = {"value": Decimal("1.20"), "date": date(2026, 9, 8),
|
||||
"when": datetime(2026, 9, 8, tzinfo=timezone.utc), "binary": b"\x00\xff"}
|
||||
graph = graph_for()
|
||||
graph.nodes[0] = graph.nodes[0].model_copy(update={"type": "source.reference", "config": {
|
||||
"source_ref": "datasource:fixture", "source_name": "records",
|
||||
}})
|
||||
source = BackendSource(node_id="source", batch=TypedBatch.from_rows([row]),
|
||||
source_ref="datasource:fixture", provider="test", fingerprint="pinned", total_rows=1)
|
||||
with patch.object(ReferenceExecutionBackend, "_execute_in_process", side_effect=AssertionError("parent evaluation")):
|
||||
result = execute_typed_graph(graph, backend="reference", sources={"source": source})
|
||||
self.assertEqual(result.rows, [{**row, "result": Decimal("1.20")}])
|
||||
self.assertEqual(result.contract.lineage.source_fingerprints[0]["fingerprint"], "pinned")
|
||||
|
||||
def test_real_pathological_regex_is_stopped_with_structured_failure(self) -> None:
|
||||
graph = graph_for("regexp_full_match(value, '(a+)+$')", [{"value": "a" * 100 + "!"}])
|
||||
started = time.monotonic()
|
||||
with self.assertRaises(BackendExecutionError) as caught:
|
||||
execute_typed_graph(graph, backend="reference", budget=ExecutionBudget(max_wall_seconds=2))
|
||||
self.assertIn(caught.exception.code, {"backend.process.timeout", "backend.process.cpu_limit"})
|
||||
self.assertLess(time.monotonic() - started, 4)
|
||||
|
||||
def test_aggregate_padding_allocation_is_contained_by_child_memory_limit(self) -> None:
|
||||
graph = graph_for("lpad(value, 900000, '0')", [{"value": "x"} for _ in range(250)])
|
||||
with self.assertRaises(BackendExecutionError) as caught:
|
||||
execute_typed_graph(graph, backend="reference", budget=ExecutionBudget(
|
||||
max_wall_seconds=5, max_memory_bytes=128 * 1024 * 1024,
|
||||
))
|
||||
self.assertEqual(caught.exception.code, "backend.process.memory_limit")
|
||||
|
||||
def test_busy_preview_is_retryable_and_never_evaluates_inline(self) -> None:
|
||||
with patch("govoplan_dataflow.backend.backends.reference.run_bounded_operation", side_effect=ProcessBudgetError("busy")):
|
||||
with self.assertRaises(PipelineExecutionError) as caught:
|
||||
_execute_pipeline_preview(graph_for(), session=None, principal=None, registry=None,
|
||||
backend="reference", row_limit=10, preview_node_id=None)
|
||||
self.assertTrue(caught.exception.retryable)
|
||||
self.assertEqual(caught.exception.diagnostics[-1].code, "backend.process.busy")
|
||||
|
||||
def test_nested_source_ids_do_not_alias_different_authorized_data(self) -> None:
|
||||
outer = graph_for()
|
||||
inner = graph_for()
|
||||
for graph, ref in ((outer, "datasource:outer"), (inner, "datasource:inner")):
|
||||
graph.nodes[0] = graph.nodes[0].model_copy(update={"type": "source.reference", "config": {
|
||||
"source_ref": ref, "source_name": "records",
|
||||
}})
|
||||
outer.nodes[1] = outer.nodes[1].model_copy(update={"type": "subflow", "config": {
|
||||
"graph": inner.model_dump(mode="python"), "parameters": {},
|
||||
}})
|
||||
def resolve(node, limit):
|
||||
return ResolvedSource(rows=({"value": node.config["source_ref"]},),
|
||||
source_ref=node.config["source_ref"], provider="test",
|
||||
fingerprint=node.config["source_ref"], total_rows=1)
|
||||
sources = _typed_backend_sources(outer, source_resolver=resolve, include_subflows=True)
|
||||
self.assertEqual({source.source_ref for source in sources.values()}, {"datasource:outer", "datasource:inner"})
|
||||
self.assertEqual(len(sources), 2)
|
||||
|
||||
def test_static_worker_documentation_is_bilingual(self) -> None:
|
||||
topic = next(item for item in get_manifest().documentation if item.id == "dataflow.reference-worker-limits")
|
||||
for body in (topic.body, topic.translations["de"]["body"]):
|
||||
self.assertIn("GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY", body)
|
||||
self.assertIn("32 MiB", body)
|
||||
self.assertEqual(set(topic.documentation_types), {"user", "admin"})
|
||||
|
||||
def test_cumulative_source_budget_stops_before_reading_further_sources(self) -> None:
|
||||
graph = graph_for()
|
||||
graph.nodes = [graph.nodes[0].model_copy(update={
|
||||
"id": f"source-{index}", "type": "source.reference", "config": {"source_ref": f"fixture:{index}"},
|
||||
}) for index in range(3)]
|
||||
calls = []
|
||||
def resolve(node, limit):
|
||||
calls.append(node.id)
|
||||
return ResolvedSource(rows=({"value": "x" * 100},), source_ref=node.config["source_ref"],
|
||||
provider="test", fingerprint="fixed", total_rows=1)
|
||||
with self.assertRaisesRegex(PipelineExecutionError, "Combined source data"):
|
||||
_typed_backend_sources(graph, source_resolver=resolve, include_subflows=True,
|
||||
_remaining_source_bytes=[200])
|
||||
self.assertEqual(calls, ["source-0", "source-1"])
|
||||
@@ -0,0 +1,194 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.core.access import PrincipalRef
|
||||
from govoplan_core.core.datasources import (
|
||||
CAPABILITY_DATASOURCE_CATALOGUE,
|
||||
DatasourceAccessError,
|
||||
DatasourceCatalogueProvider,
|
||||
DatasourceDescriptor,
|
||||
DatasourceReadRequest,
|
||||
DatasourceReadResult,
|
||||
)
|
||||
from govoplan_core.security.bounded_process import run_bounded_operation
|
||||
from govoplan_dataflow.backend.backends.reference import ReferenceExecutionBackend
|
||||
from govoplan_dataflow.backend.executor import EXECUTOR_VERSION, PipelineExecutionError
|
||||
from govoplan_dataflow.backend.graph import validate_graph
|
||||
from govoplan_dataflow.backend.schemas import GraphEdge, GraphNode, GraphPosition, PipelineGraph
|
||||
from govoplan_dataflow.backend.service import _execute_pipeline_preview
|
||||
|
||||
|
||||
OUTER_ROWS = ({"id": "outer", "amount": 15}, {"id": "outer-low", "amount": 5})
|
||||
INNER_ROWS = ({"id": "inner", "amount": 25}, {"id": "inner-low", "amount": 2})
|
||||
|
||||
|
||||
def node(node_id: str, node_type: str, config: dict) -> GraphNode:
|
||||
return GraphNode(
|
||||
id=node_id, type=node_type, label=node_id,
|
||||
position=GraphPosition(x=0, y=0), config=config,
|
||||
)
|
||||
|
||||
|
||||
def nested_source_graph() -> PipelineGraph:
|
||||
# Both external sources deliberately share their node ID and logical name.
|
||||
# The pinned subflow still has exactly one distinct inline input binding.
|
||||
nested = PipelineGraph(
|
||||
nodes=[
|
||||
node("input", "source.inline", {
|
||||
"source_name": "bound_input", "rows": [], "input_binding": True,
|
||||
}),
|
||||
node("shared", "source.reference", {
|
||||
"source_name": "records", "source_ref": {"$parameter": "source_ref"},
|
||||
"expected_fingerprint": "inner-pinned", "consistency": "frozen",
|
||||
}),
|
||||
node("union", "combine.union", {"mode": "all"}),
|
||||
node("minimum", "filter.expression", {"expression": "amount >= ${minimum}"}),
|
||||
node("output", "output", {}),
|
||||
],
|
||||
edges=[
|
||||
GraphEdge(id="input-union", source="input", target="union"),
|
||||
GraphEdge(id="shared-union", source="shared", target="union"),
|
||||
GraphEdge(id="union-minimum", source="union", target="minimum"),
|
||||
GraphEdge(id="minimum-output", source="minimum", target="output"),
|
||||
],
|
||||
)
|
||||
return PipelineGraph(
|
||||
nodes=[
|
||||
node("shared", "source.reference", {
|
||||
"source_name": "records", "source_ref": "datasource:outer",
|
||||
"expected_fingerprint": "outer-pinned", "consistency": "current",
|
||||
}),
|
||||
node("nested", "subflow", {
|
||||
"template_ref": "fixture-nested-source", "template_version": "1",
|
||||
"parameters": {"source_ref": "datasource:inner", "minimum": 10},
|
||||
"graph": nested.model_dump(mode="python"),
|
||||
}),
|
||||
node("output", "output", {}),
|
||||
],
|
||||
edges=[
|
||||
GraphEdge(id="shared-nested", source="shared", target="nested"),
|
||||
GraphEdge(id="nested-output", source="nested", target="output"),
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
class ReferenceSubflowProcessTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.graph = nested_source_graph()
|
||||
self.assertEqual([], [item.model_dump() for item in validate_graph(self.graph) if item.severity == "error"])
|
||||
self.session = Mock(spec=Session)
|
||||
self.principal = ApiPrincipal(
|
||||
principal=PrincipalRef(
|
||||
account_id="fixture-account", membership_id="fixture-membership",
|
||||
tenant_id="fixture-tenant", scopes=frozenset(),
|
||||
),
|
||||
account=object(), user=object(),
|
||||
)
|
||||
self.provider = Mock(spec=DatasourceCatalogueProvider)
|
||||
self.provider.read_datasource.side_effect = self.read_source
|
||||
self.registry = SimpleNamespace(
|
||||
has_capability=lambda name: name == CAPABILITY_DATASOURCE_CATALOGUE,
|
||||
capability=lambda name: self.provider if name == CAPABILITY_DATASOURCE_CATALOGUE else None,
|
||||
)
|
||||
self.reads: list[tuple[int, DatasourceReadRequest]] = []
|
||||
self.denied = False
|
||||
|
||||
def read_source(self, session, principal, *, request: DatasourceReadRequest) -> DatasourceReadResult:
|
||||
self.assertIs(session, self.session)
|
||||
self.assertIs(principal, self.principal)
|
||||
self.reads.append((os.getpid(), request))
|
||||
if request.datasource_ref == "datasource:inner" and self.denied:
|
||||
raise DatasourceAccessError("Current principal cannot read datasource:inner.")
|
||||
rows, fingerprint = {
|
||||
"datasource:outer": (OUTER_ROWS, "outer-pinned"),
|
||||
"datasource:inner": (INNER_ROWS, "inner-pinned"),
|
||||
}[request.datasource_ref]
|
||||
self.assertEqual(fingerprint, request.expected_fingerprint)
|
||||
return DatasourceReadResult(
|
||||
datasource=DatasourceDescriptor(
|
||||
ref=request.datasource_ref, source_name="records", name="Fixture source",
|
||||
kind="custom", mode="static", shape="tabular",
|
||||
fingerprint=fingerprint, provider="fixture-catalogue",
|
||||
),
|
||||
rows=rows, total_rows=len(rows), truncated=False,
|
||||
)
|
||||
|
||||
def preview(self, *, row_limit: int = 10):
|
||||
return _execute_pipeline_preview(
|
||||
self.graph, session=self.session, principal=self.principal,
|
||||
registry=self.registry, backend="reference", row_limit=row_limit,
|
||||
preview_node_id="nested",
|
||||
)
|
||||
|
||||
def test_nested_parameterized_sources_execute_in_real_child_without_id_aliasing(self) -> None:
|
||||
with (
|
||||
patch.object(ReferenceExecutionBackend, "_execute_in_process", side_effect=AssertionError("parent evaluation")),
|
||||
patch("govoplan_dataflow.backend.backends.reference.run_bounded_operation", wraps=run_bounded_operation) as worker,
|
||||
):
|
||||
result, version = self.preview()
|
||||
worker.assert_called_once()
|
||||
self.assertEqual(EXECUTOR_VERSION, version)
|
||||
self.assertEqual([OUTER_ROWS[0], INNER_ROWS[0]], result.rows)
|
||||
self.assertEqual(2, result.total_rows)
|
||||
self.assertFalse(result.truncated)
|
||||
self.assertEqual(2, result.input_row_count) # Root input count, not the nested binding again.
|
||||
self.assertIsNotNone(result.node_preview)
|
||||
self.assertEqual("nested", result.node_preview.node_id)
|
||||
self.assertEqual(result.rows, result.node_preview.rows)
|
||||
self.assertEqual(2, result.node_preview.total_rows)
|
||||
self.assertEqual(
|
||||
[("shared", "succeeded", 0, 2), ("nested", "succeeded", 2, 2), ("output", "succeeded", 2, 2)],
|
||||
[(item.node_id, item.status, item.input_rows, item.output_rows) for item in result.node_diagnostics],
|
||||
)
|
||||
binding_hash = hashlib.sha256(json.dumps(list(OUTER_ROWS), sort_keys=True, separators=(",", ":")).encode()).hexdigest()
|
||||
self.assertEqual([
|
||||
{"node_id": "shared", "source_ref": "datasource:outer", "source_name": "records", "kind": "datasource", "provider": "fixture-catalogue", "fingerprint": "outer-pinned", "row_count": 2, "preview_rows": 2, "truncated": False},
|
||||
{"node_id": "input", "source_name": "bound_input", "kind": "inline", "fingerprint": binding_hash, "row_count": 2, "subflow_node_id": "nested"},
|
||||
{"node_id": "shared", "source_ref": "datasource:inner", "source_name": "records", "kind": "datasource", "provider": "fixture-catalogue", "fingerprint": "inner-pinned", "row_count": 2, "preview_rows": 2, "truncated": False, "subflow_node_id": "nested"},
|
||||
], result.source_fingerprints)
|
||||
self.assertEqual(
|
||||
[("datasource:outer", "current", "outer-pinned"), ("datasource:inner", "frozen", "inner-pinned")],
|
||||
[(request.datasource_ref, request.consistency, request.expected_fingerprint) for _, request in self.reads],
|
||||
)
|
||||
self.assertTrue(all(pid == os.getpid() for pid, _ in self.reads))
|
||||
self.assertTrue(all(request.limit <= 500 and request.offset == 0 for _, request in self.reads))
|
||||
self.assertEqual([], self.session.mock_calls)
|
||||
|
||||
def test_nested_result_and_node_preview_keep_full_totals_when_output_is_bounded(self) -> None:
|
||||
result, _ = self.preview(row_limit=1)
|
||||
self.assertEqual([OUTER_ROWS[0]], result.rows)
|
||||
self.assertEqual(2, result.total_rows)
|
||||
self.assertTrue(result.truncated)
|
||||
self.assertEqual([OUTER_ROWS[0]], result.node_preview.rows)
|
||||
self.assertEqual(2, result.node_preview.total_rows)
|
||||
self.assertTrue(result.node_preview.truncated)
|
||||
self.assertEqual({"outer-pinned", "inner-pinned"}, {
|
||||
item["fingerprint"] for item in result.source_fingerprints if item["kind"] == "datasource"
|
||||
})
|
||||
self.assertEqual([], self.session.mock_calls)
|
||||
|
||||
def test_denied_nested_datasource_stops_before_worker_or_persistence(self) -> None:
|
||||
self.denied = True
|
||||
with (
|
||||
patch("govoplan_dataflow.backend.service.execute_typed_graph") as execute,
|
||||
patch("govoplan_dataflow.backend.backends.reference.run_bounded_operation") as worker,
|
||||
self.assertRaises(PipelineExecutionError) as caught,
|
||||
):
|
||||
self.preview()
|
||||
execute.assert_not_called()
|
||||
worker.assert_not_called()
|
||||
self.assertEqual("shared", caught.exception.node_id)
|
||||
self.assertEqual("Current principal cannot read datasource:inner.", str(caught.exception))
|
||||
self.assertFalse(caught.exception.retryable)
|
||||
self.assertIsInstance(caught.exception.__cause__, DatasourceAccessError)
|
||||
self.assertEqual(["datasource:outer", "datasource:inner"], [request.datasource_ref for _, request in self.reads])
|
||||
self.assertEqual([], self.session.mock_calls)
|
||||
+33
-2
@@ -123,8 +123,9 @@ def runtime_identity() -> RuntimeIdentity:
|
||||
|
||||
|
||||
class FakePublicationProvider:
|
||||
def __init__(self) -> None:
|
||||
def __init__(self, status: str = "published") -> None:
|
||||
self.requests = []
|
||||
self.status = status
|
||||
|
||||
def publish_rows(self, _session, _principal, *, request):
|
||||
self.requests.append(request)
|
||||
@@ -139,7 +140,7 @@ class FakePublicationProvider:
|
||||
)
|
||||
return DatasourcePublicationResult(
|
||||
ref="publication:publication-1",
|
||||
status="published",
|
||||
status=self.status,
|
||||
datasource=descriptor,
|
||||
materialization=DatasourceMaterialization(
|
||||
ref="materialization:materialization-1",
|
||||
@@ -539,6 +540,36 @@ class DataflowServiceTests(unittest.TestCase):
|
||||
),
|
||||
)
|
||||
|
||||
def test_publication_review_state_is_exposed_to_workflow_as_a_warning(
|
||||
self,
|
||||
) -> None:
|
||||
pipeline = self._create()
|
||||
provider = FakePublicationProvider("review_required")
|
||||
|
||||
run, _ = start_pipeline_run(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
actor_id="user-1",
|
||||
principal=principal(),
|
||||
registry=FakeRegistry(provider),
|
||||
request=DataflowRunRequest(
|
||||
pipeline_ref=f"pipeline:{pipeline.id}",
|
||||
revision=1,
|
||||
idempotency_key="review-publication",
|
||||
publication=DataflowPublicationTarget(
|
||||
name="Review output",
|
||||
source_name="review_output",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
self.assertEqual("succeeded", run.status)
|
||||
self.assertEqual(
|
||||
"publication.review_required",
|
||||
run.diagnostics[-1]["code"],
|
||||
)
|
||||
self.assertEqual("warning", run.diagnostics[-1]["severity"])
|
||||
|
||||
def test_publication_without_datasources_finishes_as_failed_run(self) -> None:
|
||||
pipeline = self._create()
|
||||
run, _ = start_pipeline_run(
|
||||
|
||||
+351
-2
@@ -25,12 +25,16 @@ from govoplan_dataflow.backend.schemas import (
|
||||
DataflowTriggerSchedule,
|
||||
PipelineCreateRequest,
|
||||
PipelineDeriveRequest,
|
||||
PipelineRebaseRequest,
|
||||
PipelineUpdateRequest,
|
||||
)
|
||||
from govoplan_dataflow.backend.service import (
|
||||
DataflowConflictError,
|
||||
DataflowValidationError,
|
||||
create_pipeline,
|
||||
derive_pipeline,
|
||||
pipeline_response,
|
||||
rebase_pipeline,
|
||||
start_pipeline_run,
|
||||
update_pipeline,
|
||||
)
|
||||
@@ -46,10 +50,87 @@ POLICY_CAPABILITY = "policy.definitionGovernance"
|
||||
AUTOMATION_CAPABILITY = "auth.automationPrincipalProvider"
|
||||
|
||||
|
||||
def sample_graph():
|
||||
def sample_graph(*, minimum: int = 10):
|
||||
from test_service import sample_graph as build_graph
|
||||
|
||||
return build_graph()
|
||||
return build_graph(minimum=minimum)
|
||||
|
||||
|
||||
def reusable_graph(*, minimum: int = 10):
|
||||
graph = sample_graph(minimum=minimum)
|
||||
return graph.model_copy(
|
||||
update={
|
||||
"nodes": [
|
||||
(
|
||||
node.model_copy(
|
||||
update={
|
||||
"config": {
|
||||
**node.config,
|
||||
"input_binding": True,
|
||||
}
|
||||
},
|
||||
deep=True,
|
||||
)
|
||||
if node.id == "source"
|
||||
else node
|
||||
)
|
||||
for node in graph.nodes
|
||||
]
|
||||
},
|
||||
deep=True,
|
||||
)
|
||||
|
||||
|
||||
def referencing_graph(
|
||||
source_ref: str,
|
||||
source_revision: int,
|
||||
*,
|
||||
omit_amount: bool = False,
|
||||
input_binding: bool = False,
|
||||
):
|
||||
graph = sample_graph()
|
||||
return graph.model_copy(
|
||||
update={
|
||||
"nodes": [
|
||||
(
|
||||
node.model_copy(
|
||||
update={
|
||||
"config": {
|
||||
**node.config,
|
||||
"rows": (
|
||||
[{"id": 1}]
|
||||
if omit_amount
|
||||
else node.config["rows"]
|
||||
),
|
||||
"input_binding": input_binding,
|
||||
}
|
||||
},
|
||||
deep=True,
|
||||
)
|
||||
if node.id == "source"
|
||||
else node.model_copy(
|
||||
update={
|
||||
"type": "subflow",
|
||||
"label": "Governed reusable flow",
|
||||
"config": {
|
||||
"template_ref": source_ref,
|
||||
"template_version": str(source_revision),
|
||||
"parameters": {},
|
||||
"graph": sample_graph(
|
||||
minimum=999
|
||||
).model_dump(mode="json"),
|
||||
},
|
||||
},
|
||||
deep=True,
|
||||
)
|
||||
if node.id == "filter"
|
||||
else node
|
||||
)
|
||||
for node in graph.nodes
|
||||
]
|
||||
},
|
||||
deep=True,
|
||||
)
|
||||
|
||||
|
||||
def principal() -> ApiPrincipal:
|
||||
@@ -482,6 +563,152 @@ class DataflowTriggerTests(unittest.TestCase):
|
||||
),
|
||||
)
|
||||
|
||||
def test_reusable_reference_is_policy_resolved_with_typed_contracts(
|
||||
self,
|
||||
) -> None:
|
||||
template = create_pipeline(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
actor_id="template-author",
|
||||
payload=PipelineCreateRequest(
|
||||
name="Typed reusable filter",
|
||||
graph=reusable_graph(minimum=10),
|
||||
definition_kind="template",
|
||||
allow_reuse=True,
|
||||
),
|
||||
)
|
||||
consumer = create_pipeline(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
actor_id="account-1",
|
||||
principal=self.actor,
|
||||
registry=self.registry,
|
||||
payload=PipelineCreateRequest(
|
||||
name="Resolved consumer",
|
||||
graph=referencing_graph(f"pipeline:{template.id}", 1),
|
||||
),
|
||||
)
|
||||
self.session.commit()
|
||||
|
||||
stored = consumer.revisions[0].graph
|
||||
subflow = next(
|
||||
node for node in stored["nodes"] if node["id"] == "filter"
|
||||
)
|
||||
self.assertEqual(template.revisions[0].content_hash, subflow["config"]["template_hash"])
|
||||
self.assertEqual(
|
||||
10,
|
||||
next(
|
||||
node
|
||||
for node in subflow["config"]["graph"]["nodes"]
|
||||
if node["id"] == "filter"
|
||||
)["config"]["value"],
|
||||
)
|
||||
self.assertEqual(
|
||||
{"id", "amount"},
|
||||
{
|
||||
field["name"]
|
||||
for field in subflow["config"]["input_schema"]
|
||||
},
|
||||
)
|
||||
self.assertEqual(
|
||||
{"id", "amount"},
|
||||
{
|
||||
field["name"]
|
||||
for field in subflow["config"]["output_schema"]
|
||||
},
|
||||
)
|
||||
self.assertTrue(
|
||||
subflow["config"]["reference_provenance"][
|
||||
"policy_decision"
|
||||
]["allowed"]
|
||||
)
|
||||
|
||||
with self.assertRaises(DataflowValidationError):
|
||||
create_pipeline(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
actor_id="account-1",
|
||||
principal=self.actor,
|
||||
registry=self.registry,
|
||||
payload=PipelineCreateRequest(
|
||||
name="Incompatible consumer",
|
||||
graph=referencing_graph(
|
||||
f"pipeline:{template.id}",
|
||||
1,
|
||||
omit_amount=True,
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
def test_reusable_reference_cycles_are_rejected_across_revisions(
|
||||
self,
|
||||
) -> None:
|
||||
left = create_pipeline(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
actor_id="account-1",
|
||||
payload=PipelineCreateRequest(
|
||||
name="Left template",
|
||||
graph=reusable_graph(),
|
||||
definition_kind="template",
|
||||
allow_reuse=True,
|
||||
),
|
||||
)
|
||||
right = create_pipeline(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
actor_id="account-1",
|
||||
payload=PipelineCreateRequest(
|
||||
name="Right template",
|
||||
graph=reusable_graph(),
|
||||
definition_kind="template",
|
||||
allow_reuse=True,
|
||||
),
|
||||
)
|
||||
self.session.flush()
|
||||
update_pipeline(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
pipeline_id=left.id,
|
||||
actor_id="account-1",
|
||||
principal=self.actor,
|
||||
registry=self.registry,
|
||||
payload=PipelineUpdateRequest(
|
||||
name=left.name,
|
||||
graph=referencing_graph(
|
||||
f"pipeline:{right.id}",
|
||||
1,
|
||||
input_binding=True,
|
||||
),
|
||||
status="draft",
|
||||
expected_revision=1,
|
||||
definition_kind="template",
|
||||
allow_reuse=True,
|
||||
),
|
||||
)
|
||||
|
||||
with self.assertRaisesRegex(DataflowConflictError, "cannot reference itself"):
|
||||
update_pipeline(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
pipeline_id=right.id,
|
||||
actor_id="account-1",
|
||||
principal=self.actor,
|
||||
registry=self.registry,
|
||||
payload=PipelineUpdateRequest(
|
||||
name=right.name,
|
||||
graph=referencing_graph(
|
||||
f"pipeline:{left.id}",
|
||||
2,
|
||||
input_binding=True,
|
||||
),
|
||||
status="draft",
|
||||
expected_revision=1,
|
||||
definition_kind="template",
|
||||
allow_reuse=True,
|
||||
),
|
||||
)
|
||||
|
||||
def test_derived_limits_cannot_be_broadened_transitively(self) -> None:
|
||||
template = create_pipeline(
|
||||
self.session,
|
||||
@@ -549,6 +776,128 @@ class DataflowTriggerTests(unittest.TestCase):
|
||||
self.assertFalse(grandchild.allow_automation)
|
||||
self.assertFalse(grandchild.inherit_to_lower_scopes)
|
||||
|
||||
def test_source_update_is_detected_and_rebased_as_reviewed_revision(
|
||||
self,
|
||||
) -> None:
|
||||
template = create_pipeline(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
actor_id="account-1",
|
||||
payload=PipelineCreateRequest(
|
||||
name="Reusable import",
|
||||
graph=sample_graph(),
|
||||
definition_kind="template",
|
||||
allow_reuse=True,
|
||||
allow_automation=True,
|
||||
),
|
||||
)
|
||||
derived = derive_pipeline(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
actor_id="account-1",
|
||||
principal=self.actor,
|
||||
registry=self.registry,
|
||||
source_pipeline_id=template.id,
|
||||
payload=PipelineDeriveRequest(
|
||||
name="Tenant import",
|
||||
allow_run=True,
|
||||
allow_automation=True,
|
||||
),
|
||||
)
|
||||
self.session.commit()
|
||||
|
||||
before = pipeline_response(
|
||||
self.session,
|
||||
derived,
|
||||
principal=self.actor,
|
||||
registry=self.registry,
|
||||
)
|
||||
self.assertTrue(before.governance.source_available)
|
||||
self.assertFalse(before.governance.update_available)
|
||||
original_child_hash = derived.revisions[0].content_hash
|
||||
|
||||
update_pipeline(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
pipeline_id=template.id,
|
||||
actor_id="template-author",
|
||||
payload=PipelineUpdateRequest(
|
||||
name=template.name,
|
||||
graph=sample_graph(minimum=20),
|
||||
status="draft",
|
||||
expected_revision=1,
|
||||
definition_kind="template",
|
||||
allow_reuse=True,
|
||||
allow_automation=True,
|
||||
),
|
||||
)
|
||||
self.session.commit()
|
||||
source_hash = template.revisions[-1].content_hash
|
||||
|
||||
available = pipeline_response(
|
||||
self.session,
|
||||
derived,
|
||||
principal=self.actor,
|
||||
registry=self.registry,
|
||||
)
|
||||
self.assertTrue(available.governance.update_available)
|
||||
self.assertEqual(2, available.governance.source_current_revision)
|
||||
self.assertEqual(source_hash, available.governance.source_current_hash)
|
||||
self.assertEqual(original_child_hash, derived.revisions[0].content_hash)
|
||||
|
||||
with self.assertRaisesRegex(DataflowConflictError, "source hash"):
|
||||
rebase_pipeline(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
pipeline_id=derived.id,
|
||||
actor_id="reviewer-1",
|
||||
principal=self.actor,
|
||||
registry=self.registry,
|
||||
payload=PipelineRebaseRequest(
|
||||
expected_revision=1,
|
||||
source_revision=2,
|
||||
source_hash="0" * 64,
|
||||
reason="Reviewed the changed filter threshold.",
|
||||
),
|
||||
)
|
||||
|
||||
rebased = rebase_pipeline(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
pipeline_id=derived.id,
|
||||
actor_id="reviewer-1",
|
||||
principal=self.actor,
|
||||
registry=self.registry,
|
||||
payload=PipelineRebaseRequest(
|
||||
expected_revision=1,
|
||||
source_revision=2,
|
||||
source_hash=source_hash,
|
||||
reason="Reviewed the changed filter threshold.",
|
||||
),
|
||||
)
|
||||
self.session.commit()
|
||||
|
||||
self.assertEqual(2, rebased.current_revision)
|
||||
self.assertEqual("draft", rebased.status)
|
||||
self.assertEqual(2, rebased.derived_from_revision)
|
||||
self.assertEqual(source_hash, rebased.derived_from_hash)
|
||||
self.assertEqual(source_hash, rebased.revisions[-1].content_hash)
|
||||
self.assertEqual(original_child_hash, rebased.revisions[0].content_hash)
|
||||
history = rebased.derivation_provenance["rebase_history"]
|
||||
self.assertEqual(1, len(history))
|
||||
self.assertEqual("reviewer-1", history[0]["rebased_by"])
|
||||
self.assertEqual(
|
||||
"Reviewed the changed filter threshold.",
|
||||
history[0]["reason"],
|
||||
)
|
||||
current = pipeline_response(
|
||||
self.session,
|
||||
rebased,
|
||||
principal=self.actor,
|
||||
registry=self.registry,
|
||||
)
|
||||
self.assertFalse(current.governance.update_available)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
+4
-3
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@govoplan/dataflow-webui",
|
||||
"version": "0.1.17",
|
||||
"version": "0.1.25",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
@@ -15,10 +15,11 @@
|
||||
},
|
||||
"scripts": {
|
||||
"typecheck": "tsc --noEmit",
|
||||
"test:structure": "node scripts/test-dataflow-page-structure.mjs"
|
||||
"test:structure": "node scripts/test-dataflow-page-structure.mjs",
|
||||
"test:save-completion": "node --test scripts/test-save-completion.mjs"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@govoplan/core-webui": "^0.1.17",
|
||||
"@govoplan/core-webui": "^0.1.46",
|
||||
"@xyflow/react": "^12.11.2",
|
||||
"lucide-react": "^1.23.0",
|
||||
"react": ">=19.2.7 <20",
|
||||
|
||||
@@ -60,11 +60,14 @@ const checks = [
|
||||
moduleEntry.indexOf("./styles/dataflow.css"),
|
||||
"XyFlow base styles before GovOPlaN overrides"
|
||||
],
|
||||
[css.includes("height: calc(100vh - 115px)"), "full-height workspace"],
|
||||
[
|
||||
page.includes('<WorkspaceFrame as="main" height="viewport"'),
|
||||
"central full-height workspace"
|
||||
],
|
||||
[css.includes(".dataflow-preview-table-wrap"), "bounded preview scrolling"],
|
||||
[
|
||||
/\.dataflow-palette-items\s*\{[^}]*grid-auto-rows:\s*max-content;[^}]*overflow-y:\s*auto;/s.test(css),
|
||||
"bounded palette scrolling"
|
||||
page.includes("<DefinitionPalette") && !css.includes(".dataflow-palette-items"),
|
||||
"centralized bounded palette scrolling"
|
||||
],
|
||||
[
|
||||
/\.dataflow-inspector-fields\s*\{[^}]*grid-auto-rows:\s*max-content;[^}]*overflow-y:\s*auto;/s.test(css),
|
||||
|
||||
Executable
+200
@@ -0,0 +1,200 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import test from "node:test";
|
||||
import vm from "node:vm";
|
||||
|
||||
const require = createRequire(new URL("../../../govoplan-core/webui/package.json", import.meta.url));
|
||||
const { transformSync } = require("esbuild");
|
||||
const page = readFileSync(new URL("../src/features/dataflow/DataflowPage.tsx", import.meta.url), "utf8");
|
||||
function loadTs(path) {
|
||||
const context = vm.createContext({ module: { exports: {} }, require: () => ({}), structuredClone });
|
||||
context.exports = context.module.exports;
|
||||
vm.runInContext(transformSync(readFileSync(new URL(path, import.meta.url), "utf8"), { loader: "ts", format: "cjs" }).code, context);
|
||||
return context.module.exports;
|
||||
}
|
||||
const { reconcilePipelineSave } = loadTs("../src/features/dataflow/saveCompletion.ts");
|
||||
const { draftFingerprint, pipelinePayload } = loadTs("../src/features/dataflow/model.ts");
|
||||
const { authAuthorityKey } = loadTs("../../../govoplan-core/webui/src/api/authAuthority.ts");
|
||||
const start = page.indexOf(" const saveDraft = useCallback(async (): Promise<boolean> => {");
|
||||
const end = page.indexOf(" }, [canEdit, draft, settings, authorityKey, authorityGeneration]);", start);
|
||||
assert.ok(start >= 0 && end > start, "exercise the actual page save closure");
|
||||
const code = transformSync(page.slice(start, end) + " }, []);\nmodule.exports = saveDraft;", { loader: "ts" }).code;
|
||||
const base = () => ({
|
||||
id: "pipeline-1", currentRevision: 1, name: "Pipeline", description: "Submitted", status: "draft",
|
||||
graph: { nodes: [{ id: "a", config: { rows: [{ id: "001", value: " text " }] } }, { id: "b" }], edges: [{ id: "a-b" }] },
|
||||
sqlText: "", editorMode: "graph", scopeType: "tenant", scopeId: "tenant-1", definitionKind: "flow",
|
||||
inheritToLowerScopes: false, allowRun: true, allowReuse: true, allowAutomation: false, governance: { actions: {} }
|
||||
});
|
||||
function harness(draft = base()) {
|
||||
const calls = [];
|
||||
const responses = [];
|
||||
const context = vm.createContext({
|
||||
module: { exports: {} }, useCallback: (value) => value, structuredClone,
|
||||
draft, canEdit: true, settings: {}, saveInFlight: { current: false },
|
||||
draftSession: { current: { generation: 1, value: draft } },
|
||||
authorityKey: "authority-A", saveContext: { current: "authority-A" }, unresolvedSaveGeneration: { current: null },
|
||||
authorityGeneration: 0, saveAuthorityEpoch: { current: { key: "authority-A", revision: 0 } },
|
||||
reconcilePipelineSave, draftFingerprint, pipelinePayload, draftFromPipeline: (value) => value,
|
||||
updateDataflowPipeline: (_settings, id, payload) => new Promise((resolve, reject) => { calls.push({ id, payload }); responses.push({ resolve, reject }); }),
|
||||
createDataflowPipeline: (_settings, payload) => new Promise((resolve, reject) => { calls.push({ payload }); responses.push({ resolve, reject }); }),
|
||||
setDraftValue: (value) => { context.draft = value; },
|
||||
setSavedDraft: (value) => { context.baseline = value; },
|
||||
setSaving: (value) => { context.saving = value; },
|
||||
setError: (value) => { context.error = value; },
|
||||
setSuccess: () => {}, setPipelines: () => {}, setSelectedNodeId: () => {}, setDiagnostics: () => {}, apiErrorMessage: String
|
||||
});
|
||||
vm.runInContext(code, context);
|
||||
return { context, calls, responses, save: context.module.exports };
|
||||
}
|
||||
function accepted(draft, revision = 2) {
|
||||
return { ...structuredClone(draft), currentRevision: revision, current_revision: revision, governance: { actions: { edit: { allowed: true } } } };
|
||||
}
|
||||
|
||||
test("edits made during an accepted save survive and navigation remains blocked until saved", async () => {
|
||||
const h = harness();
|
||||
const submitted = h.context.draft;
|
||||
const pending = h.save();
|
||||
const newerGraph = { ...submitted.graph, nodes: [...submitted.graph.nodes].reverse(), custom: { preserve: ["001", null, ""] } };
|
||||
h.context.draftSession.current.value = { ...submitted, description: "Typed during save", graph: newerGraph };
|
||||
h.responses[0].resolve(accepted(submitted));
|
||||
assert.equal(await pending, false);
|
||||
assert.equal(h.context.draft.description, "Typed during save");
|
||||
assert.equal(h.context.draft.graph, newerGraph, "graph remains atomic, including order and unknown data");
|
||||
assert.equal(h.context.baseline.description, "Submitted");
|
||||
assert.equal(h.context.draft.currentRevision, 2);
|
||||
assert.notEqual(draftFingerprint(h.context.draft), draftFingerprint(h.context.baseline));
|
||||
const second = h.save();
|
||||
assert.equal(h.calls[1].payload.expected_revision, 2, "next save uses the accepted revision, not the stale submitted one");
|
||||
assert.deepEqual(h.calls[1].payload.graph, newerGraph);
|
||||
h.responses[1].resolve(accepted(h.context.draft, 3));
|
||||
assert.equal(await second, true);
|
||||
assert.equal(draftFingerprint(h.context.draft), draftFingerprint(h.context.baseline));
|
||||
});
|
||||
|
||||
test("unchanged submitted fields accept canonical response and new identities without a duplicate create", async () => {
|
||||
const h = harness({ ...base(), id: null, currentRevision: null });
|
||||
const pending = h.save();
|
||||
assert.equal(await h.save(), false);
|
||||
assert.equal(h.calls.length, 1);
|
||||
const response = { ...accepted(h.context.draft), id: "created", name: "Server canonical name" };
|
||||
h.responses[0].resolve(response);
|
||||
assert.equal(await pending, true);
|
||||
assert.equal(h.context.draft.id, "created");
|
||||
assert.equal(h.context.draft.name, "Server canonical name");
|
||||
});
|
||||
|
||||
test("replacement draft, authority change and unmount cannot receive an old save completion", async () => {
|
||||
for (const change of ["replacement", "authority", "unmount"]) {
|
||||
const h = harness();
|
||||
const pending = h.save();
|
||||
if (change === "authority") h.context.saveContext.current = "authority-B";
|
||||
else h.context.draftSession.current.generation += 1;
|
||||
h.responses[0].resolve(accepted(h.context.draft));
|
||||
assert.equal(await pending, false);
|
||||
assert.equal(h.context.baseline, undefined);
|
||||
assert.equal(h.context.draft.currentRevision, 1);
|
||||
}
|
||||
});
|
||||
|
||||
test("harmless session/profile object refresh preserves an accepted new identity and revision", async () => {
|
||||
const h = harness({ ...base(), id: null, currentRevision: null });
|
||||
const settings = { apiBaseUrl: "/api", apiKey: "", accessToken: "" };
|
||||
const auth = {
|
||||
user: { id: "member", account_id: "account", email: "person@example.test", display_name: "Before" },
|
||||
tenant: { id: "tenant" }, scopes: ["dataflow:pipeline:write"], roles: [], groups: []
|
||||
};
|
||||
h.context.authorityKey = authAuthorityKey(auth, settings);
|
||||
h.context.saveContext.current = h.context.authorityKey;
|
||||
const pending = h.save();
|
||||
h.context.saveContext.current = authAuthorityKey({ ...structuredClone(auth), user: { ...auth.user, display_name: "After", preferred_language: "de" } }, { ...settings });
|
||||
h.responses[0].resolve({ ...accepted(h.context.draft), id: "accepted-created-id" });
|
||||
assert.equal(await pending, true);
|
||||
assert.equal(h.context.draft.id, "accepted-created-id");
|
||||
const next = h.save();
|
||||
assert.equal(h.calls[1].id, "accepted-created-id", "retry updates the accepted identity, never creates another pipeline");
|
||||
assert.equal(h.calls[1].payload.expected_revision, 2);
|
||||
h.responses[1].resolve(accepted(h.context.draft, 3));
|
||||
assert.equal(await next, true);
|
||||
});
|
||||
|
||||
test("an accepted save across a real authority change cannot be blindly retried as a duplicate create", async () => {
|
||||
const h = harness({ ...base(), id: null, currentRevision: null });
|
||||
const pending = h.save();
|
||||
h.context.saveContext.current = "authority-B";
|
||||
h.responses[0].resolve({ ...accepted(h.context.draft), id: "accepted-under-A" });
|
||||
assert.equal(await pending, false);
|
||||
h.context.authorityKey = "authority-B";
|
||||
assert.equal(await h.save(), false);
|
||||
assert.equal(h.calls.length, 1);
|
||||
assert.match(h.context.error, /Reload and review/);
|
||||
});
|
||||
|
||||
test("returning to authority A after B does not revive a stale A save completion", async () => {
|
||||
const h = harness();
|
||||
const pending = h.save();
|
||||
h.context.saveAuthorityEpoch.current = { key: "authority-A", revision: 2 };
|
||||
h.responses[0].resolve(accepted(h.context.draft));
|
||||
assert.equal(await pending, false);
|
||||
assert.equal(h.context.baseline, undefined);
|
||||
});
|
||||
|
||||
test("conflict preserves both local draft and prior revision, allowing an explicit reviewed retry", async () => {
|
||||
const h = harness();
|
||||
const original = h.context.draft;
|
||||
const pending = h.save();
|
||||
h.responses[0].reject(new Error("revision conflict"));
|
||||
assert.equal(await pending, false);
|
||||
assert.equal(h.context.draft, original);
|
||||
assert.equal(h.context.baseline, undefined);
|
||||
assert.match(h.context.error, /revision conflict/);
|
||||
assert.equal(h.context.saveInFlight.current, false);
|
||||
});
|
||||
|
||||
test("the submitted baseline is frozen even if a nested local editor mutates a shared object", async () => {
|
||||
const h = harness();
|
||||
const serverAccepted = accepted(h.context.draft);
|
||||
const pending = h.save();
|
||||
h.context.draft.graph.nodes.reverse();
|
||||
h.responses[0].resolve(serverAccepted);
|
||||
assert.equal(await pending, false);
|
||||
assert.equal(h.context.draft.graph.nodes[0].id, "b");
|
||||
assert.equal(h.context.baseline.graph.nodes[0].id, "a");
|
||||
assert.equal(h.calls[0].payload.graph.nodes[0].id, "a", "submission data is not aliased to later editor mutations");
|
||||
});
|
||||
|
||||
test("CSV imports explicitly preserve text by default; JSON payload stays independent", () => {
|
||||
assert.match(page, /\[csvValueMode, setCsvValueMode\] = useState<"text" \| "legacy_typed">\("text"\)/);
|
||||
assert.match(page, /\? \{ format, rows \}\s*: \{ format, csv_text: csvText, delimiter, csv_value_mode: csvValueMode \}/);
|
||||
assert.match(page, /setCsvValueMode\("text"\)/);
|
||||
});
|
||||
|
||||
test("source dialog sends exact CSV content and selected mode without changing JSON rows", async () => {
|
||||
const dialog = page.slice(page.indexOf("function SourceSnapshotDialog("));
|
||||
const start = dialog.indexOf(" const create = async (): Promise<boolean> => {");
|
||||
const end = dialog.indexOf("\n };", start);
|
||||
assert.ok(start >= 0 && end > start);
|
||||
const code = transformSync(dialog.slice(start, end) + "\n}; module.exports = create;", { loader: "ts" }).code;
|
||||
const csvText = 'code,value\r\n001," text "\r\n';
|
||||
for (const format of ["csv", "json"]) {
|
||||
for (const csvValueMode of ["text", "legacy_typed"]) {
|
||||
let payload;
|
||||
const context = vm.createContext({
|
||||
module: { exports: {} }, settings: {}, format, csvValueMode, csvText, delimiter: ",",
|
||||
name: "Fixture", sourceName: "fixture", description: "", rowsText: '[{"code":"001","value":" text "}]',
|
||||
isRecord: (value) => Boolean(value) && typeof value === "object" && !Array.isArray(value),
|
||||
createDataflowSourceSnapshot: async (_settings, value) => { payload = value; return {}; },
|
||||
onCreated: () => {}, setBusy: () => {}, setError: () => {}, apiErrorMessage: String
|
||||
});
|
||||
vm.runInContext(code, context);
|
||||
assert.equal(await context.module.exports(), true);
|
||||
if (format === "csv") {
|
||||
assert.equal(payload.csv_value_mode, csvValueMode);
|
||||
assert.equal(payload.csv_text, csvText);
|
||||
} else {
|
||||
assert.equal("csv_value_mode" in payload, false);
|
||||
assert.equal(JSON.stringify(payload.rows), '[{"code":"001","value":" text "}]');
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
+109
-1
@@ -95,6 +95,11 @@ export type PipelineGovernance = {
|
||||
derived_from_pipeline_id?: string | null;
|
||||
derived_from_revision?: number | null;
|
||||
derived_from_hash?: string | null;
|
||||
source_available: boolean;
|
||||
source_name?: string | null;
|
||||
source_current_revision?: number | null;
|
||||
source_current_hash?: string | null;
|
||||
update_available: boolean;
|
||||
derivation_provenance: Record<string, unknown>;
|
||||
actions: Record<string, DefinitionActionDecision>;
|
||||
};
|
||||
@@ -205,6 +210,7 @@ export type TabularSource = {
|
||||
columns: TabularSourceColumn[];
|
||||
schema_version: string;
|
||||
fingerprint: string;
|
||||
decisions_included: boolean;
|
||||
row_count?: number | null;
|
||||
byte_count?: number | null;
|
||||
updated_at?: string | null;
|
||||
@@ -217,6 +223,38 @@ export type TabularSourceCatalogue = {
|
||||
sources: TabularSource[];
|
||||
};
|
||||
|
||||
export type ReconciliationDecisionAction = "accept" | "reject" | "correct" | "defer";
|
||||
|
||||
export type ReconciliationDecision = {
|
||||
ref: string;
|
||||
id: string;
|
||||
revision: number;
|
||||
key_hash: string;
|
||||
input_hash: string;
|
||||
action: ReconciliationDecisionAction;
|
||||
reason: string;
|
||||
correction?: Record<string, unknown> | null;
|
||||
actor_ref: string;
|
||||
decided_at: string;
|
||||
};
|
||||
|
||||
export type ReconciliationDecisionSet = {
|
||||
ref: string;
|
||||
id: string;
|
||||
pipeline_id: string;
|
||||
name: string;
|
||||
node_id?: string | null;
|
||||
resource_revision: number;
|
||||
etag: string;
|
||||
fingerprint: string;
|
||||
current_decisions: ReconciliationDecision[];
|
||||
history: ReconciliationDecision[];
|
||||
created_by?: string | null;
|
||||
updated_by?: string | null;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
};
|
||||
|
||||
export type PipelineRun = {
|
||||
ref: string;
|
||||
pipeline_id: string;
|
||||
@@ -354,6 +392,7 @@ export function createDataflowSourceSnapshot(
|
||||
format: "csv";
|
||||
csv_text: string;
|
||||
delimiter: string;
|
||||
csv_value_mode?: "text" | "legacy_typed";
|
||||
}
|
||||
)
|
||||
): Promise<TabularSource> {
|
||||
@@ -363,6 +402,58 @@ export function createDataflowSourceSnapshot(
|
||||
});
|
||||
}
|
||||
|
||||
export async function listDataflowDecisionSets(
|
||||
settings: ApiSettings,
|
||||
pipelineId: string
|
||||
): Promise<ReconciliationDecisionSet[]> {
|
||||
const response = await apiFetch<{ decision_sets: ReconciliationDecisionSet[] }>(
|
||||
settings,
|
||||
`/api/v1/dataflow/pipelines/${encodeURIComponent(pipelineId)}/decision-sets`
|
||||
);
|
||||
return response.decision_sets;
|
||||
}
|
||||
|
||||
export function createDataflowDecisionSet(
|
||||
settings: ApiSettings,
|
||||
pipelineId: string,
|
||||
payload: { name: string; node_id?: string | null }
|
||||
): Promise<ReconciliationDecisionSet> {
|
||||
return apiFetch(
|
||||
settings,
|
||||
`/api/v1/dataflow/pipelines/${encodeURIComponent(pipelineId)}/decision-sets`,
|
||||
{ method: "POST", body: JSON.stringify(payload) }
|
||||
);
|
||||
}
|
||||
|
||||
export function getDataflowDecisionSet(
|
||||
settings: ApiSettings,
|
||||
decisionSetId: string
|
||||
): Promise<ReconciliationDecisionSet> {
|
||||
return apiFetch(
|
||||
settings,
|
||||
`/api/v1/dataflow/decision-sets/${encodeURIComponent(decisionSetId)}`
|
||||
);
|
||||
}
|
||||
|
||||
export function recordDataflowDecision(
|
||||
settings: ApiSettings,
|
||||
decisionSetId: string,
|
||||
payload: {
|
||||
base_revision: number;
|
||||
key_hash: string;
|
||||
input_hash: string;
|
||||
action: ReconciliationDecisionAction;
|
||||
reason: string;
|
||||
correction?: Record<string, unknown> | null;
|
||||
}
|
||||
): Promise<ReconciliationDecisionSet> {
|
||||
return apiFetch(
|
||||
settings,
|
||||
`/api/v1/dataflow/decision-sets/${encodeURIComponent(decisionSetId)}/decisions`,
|
||||
{ method: "POST", body: JSON.stringify(payload) }
|
||||
);
|
||||
}
|
||||
|
||||
export async function listDataflowPipelines(settings: ApiSettings): Promise<Pipeline[]> {
|
||||
const response = await apiFetch<{ pipelines: Pipeline[] }>(settings, "/api/v1/dataflow/pipelines");
|
||||
return response.pipelines;
|
||||
@@ -420,6 +511,23 @@ export function deriveDataflowPipeline(
|
||||
);
|
||||
}
|
||||
|
||||
export function rebaseDataflowPipeline(
|
||||
settings: ApiSettings,
|
||||
pipelineId: string,
|
||||
payload: {
|
||||
expected_revision: number;
|
||||
source_revision: number;
|
||||
source_hash: string;
|
||||
reason: string;
|
||||
}
|
||||
): Promise<Pipeline> {
|
||||
return apiFetch(
|
||||
settings,
|
||||
`/api/v1/dataflow/pipelines/${encodeURIComponent(pipelineId)}/rebase`,
|
||||
{ method: "POST", body: JSON.stringify(payload) }
|
||||
);
|
||||
}
|
||||
|
||||
export function dataflowScopeReferenceProvider(
|
||||
settings: ApiSettings,
|
||||
scopeType: "user" | "group"
|
||||
@@ -479,7 +587,7 @@ export function deleteDataflowTrigger(
|
||||
|
||||
export function validateDataflowPipeline(
|
||||
settings: ApiSettings,
|
||||
payload: { graph?: PipelineGraph; sql_text?: string; source_nodes?: PipelineGraphNode[] }
|
||||
payload: { pipeline_id?: string | null; graph?: PipelineGraph; sql_text?: string; source_nodes?: PipelineGraphNode[] }
|
||||
): Promise<PipelineValidation> {
|
||||
return apiFetch<PipelineValidation>(settings, "/api/v1/dataflow/validate", {
|
||||
method: "POST",
|
||||
|
||||
@@ -14,6 +14,7 @@ import {
|
||||
type Edge,
|
||||
type ReactFlowInstance
|
||||
} from "@xyflow/react";
|
||||
import { StatePanel } from "@govoplan/core-webui";
|
||||
import { definitionConnectionError } from "@govoplan/core-webui/definition-graph";
|
||||
import type {
|
||||
DataflowDiagnostic,
|
||||
@@ -226,7 +227,7 @@ export default function DataflowCanvas({
|
||||
return (
|
||||
<div
|
||||
ref={canvasRef}
|
||||
className="dataflow-canvas"
|
||||
className="definition-graph-canvas dataflow-canvas"
|
||||
tabIndex={0}
|
||||
aria-label="Dataflow graph canvas"
|
||||
onDragOver={(event) => {
|
||||
@@ -360,7 +361,7 @@ export default function DataflowCanvas({
|
||||
<Controls showInteractive={false} />
|
||||
</ReactFlow>
|
||||
{!graph.nodes.length ? (
|
||||
<div className="dataflow-canvas-empty">Drop a source here</div>
|
||||
<StatePanel className="definition-graph-canvas-empty" size="fill" description="Drop a source here" />
|
||||
) : null}
|
||||
</div>
|
||||
);
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { Handle, Position, type Node, type NodeProps } from "@xyflow/react";
|
||||
import { CountBadge, DefinitionNodeIcon } from "@govoplan/core-webui";
|
||||
import type { NodeTypeDefinition } from "../../api/dataflow";
|
||||
import { dataflowNodeIcon } from "./nodeIcons";
|
||||
|
||||
@@ -31,7 +32,7 @@ export default function DataflowNode({ data, selected }: NodeProps<DataflowFlowN
|
||||
id={port.id}
|
||||
type="target"
|
||||
position={Position.Left}
|
||||
className="dataflow-node-handle dataflow-node-handle-input"
|
||||
className="definition-node-handle dataflow-node-handle-input"
|
||||
style={{ top: portPosition(index, data.definition.input_ports.length) }}
|
||||
/>
|
||||
))}
|
||||
@@ -46,15 +47,15 @@ export default function DataflowNode({ data, selected }: NodeProps<DataflowFlowN
|
||||
</span>
|
||||
))
|
||||
: null}
|
||||
<span className="dataflow-node-icon" aria-hidden="true">
|
||||
<DefinitionNodeIcon aria-hidden="true">
|
||||
<Icon size={17} strokeWidth={1.8} />
|
||||
</span>
|
||||
</DefinitionNodeIcon>
|
||||
<span className="dataflow-node-copy">
|
||||
<strong>{data.label}</strong>
|
||||
<small>{data.definition.label}</small>
|
||||
</span>
|
||||
{typeof data.outputRows === "number" ? (
|
||||
<span className="dataflow-node-count">{data.outputRows}</span>
|
||||
<CountBadge tone="neutral" size="compact">{data.outputRows}</CountBadge>
|
||||
) : null}
|
||||
{data.definition.output_ports.map((port, index) => (
|
||||
<Handle
|
||||
@@ -62,7 +63,7 @@ export default function DataflowNode({ data, selected }: NodeProps<DataflowFlowN
|
||||
id={port.id}
|
||||
type="source"
|
||||
position={Position.Right}
|
||||
className="dataflow-node-handle dataflow-node-handle-output"
|
||||
className="definition-node-handle dataflow-node-handle-output"
|
||||
style={{ top: portPosition(index, data.definition.output_ports.length) }}
|
||||
/>
|
||||
))}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,12 +1,15 @@
|
||||
import { useEffect, useState, type ComponentProps } from "react";
|
||||
import { Play, Trash2 } from "lucide-react";
|
||||
import {
|
||||
ActionToolbar,
|
||||
DismissibleAlert,
|
||||
FormField as CoreFormField,
|
||||
IconButton
|
||||
IconButton,
|
||||
StatePanel
|
||||
} from "@govoplan/core-webui";
|
||||
import type {
|
||||
NodeTypeDefinition,
|
||||
Pipeline,
|
||||
PipelineGraphNode,
|
||||
TabularSource
|
||||
} from "../../api/dataflow";
|
||||
@@ -14,7 +17,7 @@ import { DATAFLOW_NODE_DOCUMENTATION } from "./interfacePatterns";
|
||||
|
||||
type NodeFormFieldProps = ComponentProps<typeof CoreFormField>;
|
||||
|
||||
function FormField({ documentation, ...props }: NodeFormFieldProps) {
|
||||
function NodeFormField({ documentation, ...props }: NodeFormFieldProps) {
|
||||
return (
|
||||
<CoreFormField
|
||||
{...props}
|
||||
@@ -26,6 +29,7 @@ function FormField({ documentation, ...props }: NodeFormFieldProps) {
|
||||
type NodeInspectorProps = {
|
||||
node: PipelineGraphNode | null;
|
||||
nodeLibrary: NodeTypeDefinition[];
|
||||
reusablePipelines: Pipeline[];
|
||||
sources: TabularSource[];
|
||||
sourceCatalogueAvailable: boolean;
|
||||
readOnly: boolean;
|
||||
@@ -39,6 +43,7 @@ type NodeInspectorProps = {
|
||||
export default function NodeInspector({
|
||||
node,
|
||||
nodeLibrary,
|
||||
reusablePipelines,
|
||||
sources,
|
||||
sourceCatalogueAvailable,
|
||||
readOnly,
|
||||
@@ -55,7 +60,6 @@ export default function NodeInspector({
|
||||
const [rankSortText, setRankSortText] = useState("");
|
||||
const [rulesText, setRulesText] = useState("");
|
||||
const [parametersText, setParametersText] = useState("");
|
||||
const [subflowGraphText, setSubflowGraphText] = useState("");
|
||||
const [localError, setLocalError] = useState("");
|
||||
|
||||
useEffect(() => {
|
||||
@@ -66,22 +70,24 @@ export default function NodeInspector({
|
||||
setRankSortText(node ? sortFieldsToText(node.config.order_by) : "");
|
||||
setRulesText(node ? JSON.stringify(node.config.rules ?? [], null, 2) : "");
|
||||
setParametersText(node ? JSON.stringify(node.config.parameters ?? {}, null, 2) : "");
|
||||
setSubflowGraphText(node ? JSON.stringify(node.config.graph ?? {}, null, 2) : "");
|
||||
setLocalError("");
|
||||
}, [node?.id]);
|
||||
|
||||
if (!node) {
|
||||
return (
|
||||
<aside className="dataflow-inspector" aria-label="Node inspector">
|
||||
<div className="dataflow-panel-heading">
|
||||
<ActionToolbar surface="section-header" className="dataflow-panel-heading">
|
||||
<strong>Inspector</strong>
|
||||
</div>
|
||||
<div className="dataflow-inspector-empty">No node selected</div>
|
||||
</ActionToolbar>
|
||||
<StatePanel size="compact" description="No node selected" />
|
||||
</aside>
|
||||
);
|
||||
}
|
||||
|
||||
const definition = nodeLibrary.find((item) => item.type === node.type);
|
||||
const selectedReusable = reusablePipelines.find(
|
||||
(item) => `pipeline:${item.id}` === textValue(node.config.template_ref)
|
||||
);
|
||||
const updateConfig = (patch: Record<string, unknown>) => {
|
||||
onChange({ ...node, config: { ...node.config, ...patch } });
|
||||
};
|
||||
@@ -161,7 +167,7 @@ export default function NodeInspector({
|
||||
|
||||
return (
|
||||
<aside className="dataflow-inspector" aria-label="Node inspector">
|
||||
<div className="dataflow-panel-heading">
|
||||
<ActionToolbar surface="section-header" className="dataflow-panel-heading">
|
||||
<span>
|
||||
<strong>Inspector</strong>
|
||||
<small>{definition?.label ?? node.type}</small>
|
||||
@@ -176,38 +182,40 @@ export default function NodeInspector({
|
||||
/>
|
||||
<IconButton
|
||||
label="Delete node"
|
||||
helpContextId="dataflow.action.delete"
|
||||
helpModuleId="dataflow"
|
||||
icon={<Trash2 size={16} />}
|
||||
variant="danger"
|
||||
onClick={() => onDelete(node.id)}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</span>
|
||||
</div>
|
||||
</ActionToolbar>
|
||||
<div className="dataflow-inspector-fields">
|
||||
{localError ? (
|
||||
<DismissibleAlert tone="danger" resetKey={localError}>
|
||||
{localError}
|
||||
</DismissibleAlert>
|
||||
) : null}
|
||||
<FormField label="Name">
|
||||
<NodeFormField label="Name">
|
||||
<input
|
||||
value={node.label}
|
||||
onChange={(event) => onChange({ ...node, label: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
{node.type.startsWith("source.") ? (
|
||||
<FormField label="Logical source name">
|
||||
<NodeFormField label="Logical source name">
|
||||
<input
|
||||
value={textValue(node.config.source_name)}
|
||||
onChange={(event) => updateConfig({ source_name: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
) : null}
|
||||
{node.type === "source.reference" ? (
|
||||
<>
|
||||
<FormField label="Source">
|
||||
<NodeFormField label="Source">
|
||||
<select
|
||||
value={textValue(node.config.source_ref)}
|
||||
onChange={(event) => {
|
||||
@@ -237,8 +245,8 @@ export default function NodeInspector({
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
</FormField>
|
||||
<FormField label="State">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="State">
|
||||
<select
|
||||
value={textValue(node.config.consistency) || "current"}
|
||||
onChange={(event) => updateConfig({ consistency: event.target.value })}
|
||||
@@ -248,20 +256,21 @@ export default function NodeInspector({
|
||||
<option value="live">Live</option>
|
||||
<option value="frozen">Latest frozen</option>
|
||||
</select>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
{textValue(node.config.expected_fingerprint) ? (
|
||||
<FormField label="Pinned fingerprint">
|
||||
<NodeFormField label="Pinned fingerprint">
|
||||
<input
|
||||
value={textValue(node.config.expected_fingerprint)}
|
||||
readOnly
|
||||
title={textValue(node.config.expected_fingerprint)}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
) : null}
|
||||
</>
|
||||
) : null}
|
||||
{node.type === "source.inline" ? (
|
||||
<FormField label="Rows">
|
||||
<>
|
||||
<NodeFormField label="Rows">
|
||||
<textarea
|
||||
className="dataflow-json-editor"
|
||||
value={rowsText}
|
||||
@@ -270,18 +279,34 @@ export default function NodeInspector({
|
||||
spellCheck={false}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
<NodeFormField
|
||||
label="Reusable input binding"
|
||||
help="A reusable definition must mark exactly one typed inline source as the rows supplied by its caller."
|
||||
interfaceId="dataflow.field.reusable-input-binding"
|
||||
helpContextId="dataflow.field.reusable-input-binding"
|
||||
helpModuleId="dataflow"
|
||||
helpTopicId="dataflow.reference.nodes-and-expressions"
|
||||
>
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={node.config.input_binding === true}
|
||||
onChange={(event) => updateConfig({ input_binding: event.target.checked })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</NodeFormField>
|
||||
</>
|
||||
) : null}
|
||||
{node.type === "filter" ? (
|
||||
<>
|
||||
<FormField label="Column">
|
||||
<NodeFormField label="Column">
|
||||
<input
|
||||
value={textValue(node.config.column)}
|
||||
onChange={(event) => updateConfig({ column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Operator">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Operator">
|
||||
<select
|
||||
value={textValue(node.config.operator) || "eq"}
|
||||
onChange={(event) => updateConfig({ operator: event.target.value })}
|
||||
@@ -297,20 +322,20 @@ export default function NodeInspector({
|
||||
<option value="is_null">is null</option>
|
||||
<option value="not_null">is not null</option>
|
||||
</select>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
{!["is_null", "not_null"].includes(textValue(node.config.operator)) ? (
|
||||
<FormField label="Value">
|
||||
<NodeFormField label="Value">
|
||||
<input
|
||||
value={displayScalar(node.config.value)}
|
||||
onChange={(event) => updateConfig({ value: parseScalar(event.target.value) })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
) : null}
|
||||
</>
|
||||
) : null}
|
||||
{node.type === "filter.expression" ? (
|
||||
<FormField label="Expression">
|
||||
<NodeFormField label="Expression">
|
||||
<textarea
|
||||
className="dataflow-expression-editor"
|
||||
value={textValue(node.config.expression)}
|
||||
@@ -318,20 +343,20 @@ export default function NodeInspector({
|
||||
spellCheck={false}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
) : null}
|
||||
{node.type === "distinct" ? (
|
||||
<FormField label="Key columns">
|
||||
<NodeFormField label="Key columns">
|
||||
<input
|
||||
value={stringList(node.config.columns).join(", ")}
|
||||
onChange={(event) => updateConfig({ columns: commaList(event.target.value) })}
|
||||
placeholder="All columns"
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
) : null}
|
||||
{node.type === "combine.union" ? (
|
||||
<FormField label="Duplicates">
|
||||
<NodeFormField label="Duplicates">
|
||||
<select
|
||||
value={textValue(node.config.mode) || "all"}
|
||||
onChange={(event) => updateConfig({ mode: event.target.value })}
|
||||
@@ -340,11 +365,11 @@ export default function NodeInspector({
|
||||
<option value="all">Keep all rows</option>
|
||||
<option value="distinct">Remove duplicates</option>
|
||||
</select>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
) : null}
|
||||
{node.type === "combine.join" ? (
|
||||
<>
|
||||
<FormField label="Join type">
|
||||
<NodeFormField label="Join type">
|
||||
<select
|
||||
value={textValue(node.config.join_type) || "inner"}
|
||||
onChange={(event) => updateConfig({ join_type: event.target.value })}
|
||||
@@ -357,51 +382,51 @@ export default function NodeInspector({
|
||||
<option value="semi">Left rows with a match</option>
|
||||
<option value="anti">Left rows without a match</option>
|
||||
</select>
|
||||
</FormField>
|
||||
<FormField label="Left keys">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Left keys">
|
||||
<input
|
||||
value={stringList(node.config.left_keys).join(", ")}
|
||||
onChange={(event) => updateConfig({ left_keys: commaList(event.target.value) })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Right keys">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Right keys">
|
||||
<input
|
||||
value={stringList(node.config.right_keys).join(", ")}
|
||||
onChange={(event) => updateConfig({ right_keys: commaList(event.target.value) })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
{!["semi", "anti"].includes(textValue(node.config.join_type)) ? (
|
||||
<FormField label="Right-column prefix">
|
||||
<NodeFormField label="Right-column prefix">
|
||||
<input
|
||||
value={textValue(node.config.right_prefix)}
|
||||
onChange={(event) => updateConfig({ right_prefix: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
) : null}
|
||||
</>
|
||||
) : null}
|
||||
{node.type === "select" ? (
|
||||
<FormField label="Columns">
|
||||
<NodeFormField label="Columns">
|
||||
<input
|
||||
value={selectFieldsToText(node.config.fields)}
|
||||
onChange={(event) => updateConfig({ fields: selectFieldsFromText(event.target.value) })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
) : null}
|
||||
{node.type === "aggregate" ? (
|
||||
<>
|
||||
<FormField label="Group by">
|
||||
<NodeFormField label="Group by">
|
||||
<input
|
||||
value={stringList(node.config.group_by).join(", ")}
|
||||
onChange={(event) => updateConfig({ group_by: commaList(event.target.value) })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Aggregates">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Aggregates">
|
||||
<textarea
|
||||
className="dataflow-expression-editor"
|
||||
value={aggregateText}
|
||||
@@ -410,19 +435,19 @@ export default function NodeInspector({
|
||||
spellCheck={false}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
</>
|
||||
) : null}
|
||||
{node.type === "derive" ? (
|
||||
<>
|
||||
<FormField label="Output column">
|
||||
<NodeFormField label="Output column">
|
||||
<input
|
||||
value={textValue(node.config.target_column)}
|
||||
onChange={(event) => updateConfig({ target_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Operation">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Operation">
|
||||
<select
|
||||
value={textValue(node.config.operation) || "copy"}
|
||||
onChange={(event) => updateConfig({ operation: event.target.value })}
|
||||
@@ -439,35 +464,35 @@ export default function NodeInspector({
|
||||
<option value="multiply">Multiply</option>
|
||||
<option value="divide">Divide</option>
|
||||
</select>
|
||||
</FormField>
|
||||
<FormField label="Source columns">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Source columns">
|
||||
<input
|
||||
value={stringList(node.config.source_columns).join(", ")}
|
||||
onChange={(event) => updateConfig({ source_columns: commaList(event.target.value) })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
{textValue(node.config.operation) === "concat" ? (
|
||||
<FormField label="Separator">
|
||||
<NodeFormField label="Separator">
|
||||
<input
|
||||
value={textValue(node.config.separator)}
|
||||
onChange={(event) => updateConfig({ separator: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
) : null}
|
||||
</>
|
||||
) : null}
|
||||
{node.type === "expression" ? (
|
||||
<>
|
||||
<FormField label="Output column">
|
||||
<NodeFormField label="Output column">
|
||||
<input
|
||||
value={textValue(node.config.target_column)}
|
||||
onChange={(event) => updateConfig({ target_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Expression">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Expression">
|
||||
<textarea
|
||||
className="dataflow-expression-editor"
|
||||
value={textValue(node.config.expression)}
|
||||
@@ -475,8 +500,8 @@ export default function NodeInspector({
|
||||
spellCheck={false}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Expected type">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Expected type">
|
||||
<select
|
||||
value={textValue(node.config.result_type) || "unknown"}
|
||||
onChange={(event) => updateConfig({ result_type: event.target.value })}
|
||||
@@ -490,11 +515,11 @@ export default function NodeInspector({
|
||||
<option value="date">Date</option>
|
||||
<option value="datetime">Date and time</option>
|
||||
</select>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
</>
|
||||
) : null}
|
||||
{node.type === "calculate" ? (
|
||||
<FormField label="Calculated columns">
|
||||
<NodeFormField label="Calculated columns">
|
||||
<textarea
|
||||
className="dataflow-expression-editor"
|
||||
value={calculationText}
|
||||
@@ -503,25 +528,25 @@ export default function NodeInspector({
|
||||
spellCheck={false}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
) : null}
|
||||
{node.type === "convert" ? (
|
||||
<>
|
||||
<FormField label="Source column">
|
||||
<NodeFormField label="Source column">
|
||||
<input
|
||||
value={textValue(node.config.source_column)}
|
||||
onChange={(event) => updateConfig({ source_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Output column">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Output column">
|
||||
<input
|
||||
value={textValue(node.config.target_column)}
|
||||
onChange={(event) => updateConfig({ target_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Data type">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Data type">
|
||||
<select
|
||||
value={textValue(node.config.target_type) || "string"}
|
||||
onChange={(event) => updateConfig({ target_type: event.target.value })}
|
||||
@@ -534,8 +559,8 @@ export default function NodeInspector({
|
||||
<option value="date">Date</option>
|
||||
<option value="datetime">Date and time</option>
|
||||
</select>
|
||||
</FormField>
|
||||
<FormField label="Conversion error">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Conversion error">
|
||||
<select
|
||||
value={textValue(node.config.on_error) || "fail"}
|
||||
onChange={(event) => updateConfig({ on_error: event.target.value })}
|
||||
@@ -545,26 +570,26 @@ export default function NodeInspector({
|
||||
<option value="null">Use null</option>
|
||||
<option value="keep">Keep original</option>
|
||||
</select>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
</>
|
||||
) : null}
|
||||
{node.type === "replace" ? (
|
||||
<>
|
||||
<FormField label="Source column">
|
||||
<NodeFormField label="Source column">
|
||||
<input
|
||||
value={textValue(node.config.source_column)}
|
||||
onChange={(event) => updateConfig({ source_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Output column">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Output column">
|
||||
<input
|
||||
value={textValue(node.config.target_column)}
|
||||
onChange={(event) => updateConfig({ target_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Mode">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Mode">
|
||||
<select
|
||||
value={textValue(node.config.mode) || "exact"}
|
||||
onChange={(event) => updateConfig({ mode: event.target.value })}
|
||||
@@ -573,25 +598,25 @@ export default function NodeInspector({
|
||||
<option value="exact">Exact value</option>
|
||||
<option value="text">Text fragment</option>
|
||||
</select>
|
||||
</FormField>
|
||||
<FormField label="Find">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Find">
|
||||
<input
|
||||
value={displayScalar(node.config.find)}
|
||||
onChange={(event) => updateConfig({ find: parseScalar(event.target.value) })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Replacement">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Replacement">
|
||||
<input
|
||||
value={displayScalar(node.config.replacement)}
|
||||
onChange={(event) => updateConfig({ replacement: parseScalar(event.target.value) })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
</>
|
||||
) : null}
|
||||
{node.type === "sort" ? (
|
||||
<FormField label="Sort fields">
|
||||
<NodeFormField label="Sort fields">
|
||||
<textarea
|
||||
className="dataflow-expression-editor"
|
||||
value={sortText}
|
||||
@@ -600,11 +625,11 @@ export default function NodeInspector({
|
||||
spellCheck={false}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
) : null}
|
||||
{node.type === "window.rank" ? (
|
||||
<>
|
||||
<FormField label="Method">
|
||||
<NodeFormField label="Method">
|
||||
<select
|
||||
value={textValue(node.config.method) || "row_number"}
|
||||
onChange={(event) => updateConfig({ method: event.target.value })}
|
||||
@@ -614,22 +639,22 @@ export default function NodeInspector({
|
||||
<option value="rank">Rank with gaps</option>
|
||||
<option value="dense_rank">Dense rank</option>
|
||||
</select>
|
||||
</FormField>
|
||||
<FormField label="Output column">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Output column">
|
||||
<input
|
||||
value={textValue(node.config.target_column)}
|
||||
onChange={(event) => updateConfig({ target_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Partition by">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Partition by">
|
||||
<input
|
||||
value={stringList(node.config.partition_by).join(", ")}
|
||||
onChange={(event) => updateConfig({ partition_by: commaList(event.target.value) })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Order by">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Order by">
|
||||
<textarea
|
||||
className="dataflow-expression-editor"
|
||||
value={rankSortText}
|
||||
@@ -638,11 +663,11 @@ export default function NodeInspector({
|
||||
spellCheck={false}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
</>
|
||||
) : null}
|
||||
{node.type === "limit" ? (
|
||||
<FormField label="Maximum rows">
|
||||
<NodeFormField label="Maximum rows">
|
||||
<input
|
||||
type="number"
|
||||
min={1}
|
||||
@@ -651,11 +676,11 @@ export default function NodeInspector({
|
||||
onChange={(event) => updateConfig({ count: Number(event.target.value) })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
) : null}
|
||||
{node.type === "quality.rules" ? (
|
||||
<>
|
||||
<FormField label="Rules">
|
||||
<NodeFormField label="Rules">
|
||||
<textarea
|
||||
className="dataflow-json-editor"
|
||||
value={rulesText}
|
||||
@@ -664,8 +689,8 @@ export default function NodeInspector({
|
||||
spellCheck={false}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Invalid rows">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Invalid rows">
|
||||
<select
|
||||
value={textValue(node.config.action) || "annotate"}
|
||||
onChange={(event) => updateConfig({ action: event.target.value })}
|
||||
@@ -675,26 +700,26 @@ export default function NodeInspector({
|
||||
<option value="drop">Drop</option>
|
||||
<option value="fail">Stop</option>
|
||||
</select>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
</>
|
||||
) : null}
|
||||
{node.type === "reconcile.compare" ? (
|
||||
<>
|
||||
<FormField label="Expected keys">
|
||||
<NodeFormField label="Expected keys">
|
||||
<input
|
||||
value={stringList(node.config.left_keys).join(", ")}
|
||||
onChange={(event) => updateConfig({ left_keys: commaList(event.target.value) })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Observed keys">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Observed keys">
|
||||
<input
|
||||
value={stringList(node.config.right_keys).join(", ")}
|
||||
onChange={(event) => updateConfig({ right_keys: commaList(event.target.value) })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Compared columns">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Compared columns">
|
||||
<input
|
||||
value={comparisonFieldsToText(node.config.compare_columns)}
|
||||
onChange={(event) => updateConfig({
|
||||
@@ -702,100 +727,143 @@ export default function NodeInspector({
|
||||
})}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Observed prefix">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Observed prefix">
|
||||
<input
|
||||
value={textValue(node.config.right_prefix)}
|
||||
onChange={(event) => updateConfig({ right_prefix: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
</>
|
||||
) : null}
|
||||
{node.type === "reconcile.decisions" ? (
|
||||
<>
|
||||
<FormField label="Decision key hash column">
|
||||
<NodeFormField label="Decision key hash column">
|
||||
<input
|
||||
value={textValue(node.config.decision_key_column)}
|
||||
onChange={(event) => updateConfig({ decision_key_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Decision input hash column">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Decision input hash column">
|
||||
<input
|
||||
value={textValue(node.config.decision_input_column)}
|
||||
onChange={(event) => updateConfig({ decision_input_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Decision reference column">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Decision reference column">
|
||||
<input
|
||||
value={textValue(node.config.decision_ref_column)}
|
||||
onChange={(event) => updateConfig({ decision_ref_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Action column">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Action column">
|
||||
<input
|
||||
value={textValue(node.config.action_column)}
|
||||
onChange={(event) => updateConfig({ action_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Actor reference column">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Actor reference column">
|
||||
<input
|
||||
value={textValue(node.config.actor_column)}
|
||||
onChange={(event) => updateConfig({ actor_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Decision time column">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Decision time column">
|
||||
<input
|
||||
value={textValue(node.config.decided_at_column)}
|
||||
onChange={(event) => updateConfig({ decided_at_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Reason column">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Reason column">
|
||||
<input
|
||||
value={textValue(node.config.reason_column)}
|
||||
onChange={(event) => updateConfig({ reason_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Correction column">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Correction column">
|
||||
<input
|
||||
value={textValue(node.config.correction_column)}
|
||||
onChange={(event) => updateConfig({ correction_column: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Allowed actions">
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Allowed actions">
|
||||
<input
|
||||
value={stringList(node.config.allowed_actions).join(", ")}
|
||||
onChange={(event) => updateConfig({ allowed_actions: commaList(event.target.value) })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
</>
|
||||
) : null}
|
||||
{node.type === "subflow" ? (
|
||||
<>
|
||||
<FormField label="Template reference">
|
||||
<input
|
||||
<NodeFormField
|
||||
label="Reusable definition"
|
||||
interfaceId="dataflow.field.subflow-reference"
|
||||
helpContextId="dataflow.field.subflow-reference"
|
||||
helpModuleId="dataflow"
|
||||
helpTopicId="dataflow.reference.nodes-and-expressions"
|
||||
>
|
||||
<select
|
||||
value={textValue(node.config.template_ref)}
|
||||
onChange={(event) => updateConfig({ template_ref: event.target.value })}
|
||||
onChange={(event) => {
|
||||
const selected = reusablePipelines.find(
|
||||
(item) => `pipeline:${item.id}` === event.target.value
|
||||
);
|
||||
updateConfig({
|
||||
template_ref: event.target.value,
|
||||
template_version: selected ? String(selected.current_revision) : "",
|
||||
template_hash: "",
|
||||
graph: { schema_version: 1, nodes: [], edges: [] },
|
||||
input_schema: [],
|
||||
output_schema: []
|
||||
});
|
||||
}}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Template version">
|
||||
<input
|
||||
>
|
||||
<option value="">Choose a reusable definition</option>
|
||||
{reusablePipelines.map((pipeline) => (
|
||||
<option key={pipeline.id} value={`pipeline:${pipeline.id}`}>
|
||||
{pipeline.name} · revision {pipeline.current_revision}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
</NodeFormField>
|
||||
<NodeFormField
|
||||
label="Template version"
|
||||
interfaceId="dataflow.field.subflow-revision"
|
||||
helpContextId="dataflow.field.subflow-revision"
|
||||
helpModuleId="dataflow"
|
||||
helpTopicId="dataflow.reference.nodes-and-expressions"
|
||||
>
|
||||
<select
|
||||
value={textValue(node.config.template_version)}
|
||||
onChange={(event) => updateConfig({ template_version: event.target.value })}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Parameters">
|
||||
>
|
||||
{textValue(node.config.template_version)
|
||||
&& textValue(node.config.template_version) !== String(selectedReusable?.current_revision ?? "") ? (
|
||||
<option value={textValue(node.config.template_version)}>
|
||||
Pinned revision {textValue(node.config.template_version)}
|
||||
</option>
|
||||
) : null}
|
||||
{selectedReusable ? (
|
||||
<option value={String(selectedReusable.current_revision)}>
|
||||
Current revision {selectedReusable.current_revision}
|
||||
</option>
|
||||
) : null}
|
||||
</select>
|
||||
</NodeFormField>
|
||||
<NodeFormField label="Parameters">
|
||||
<textarea
|
||||
className="dataflow-json-editor"
|
||||
value={parametersText}
|
||||
@@ -804,17 +872,14 @@ export default function NodeInspector({
|
||||
spellCheck={false}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label="Pinned graph">
|
||||
<textarea
|
||||
className="dataflow-json-editor"
|
||||
value={subflowGraphText}
|
||||
onChange={(event) => setSubflowGraphText(event.target.value)}
|
||||
onBlur={() => commitJsonConfig("graph", subflowGraphText, "object")}
|
||||
spellCheck={false}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
</FormField>
|
||||
</NodeFormField>
|
||||
{Array.isArray(node.config.input_schema) && Array.isArray(node.config.output_schema) ? (
|
||||
<NodeFormField label="Pinned contracts">
|
||||
<code>
|
||||
{node.config.input_schema.length} input · {node.config.output_schema.length} output fields
|
||||
</code>
|
||||
</NodeFormField>
|
||||
) : null}
|
||||
</>
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
@@ -272,7 +272,7 @@ export const FALLBACK_NODE_LIBRARY: NodeTypeDefinition[] = [
|
||||
"transform",
|
||||
"Transform",
|
||||
"Reusable subflow",
|
||||
"Run a pinned parameterized template snapshot.",
|
||||
"Run a Policy-authorized, server-resolved immutable definition revision.",
|
||||
"boxes",
|
||||
input,
|
||||
output,
|
||||
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
import type { PipelineDraft } from "./model";
|
||||
|
||||
/** Reconcile one accepted save, never structurally merge/reorder graph data.
|
||||
* Fields edited since submission stay local; identity, revision and authority
|
||||
* always come from the accepted server response.
|
||||
*/
|
||||
export function reconcilePipelineSave(
|
||||
submitted: PipelineDraft, current: PipelineDraft, accepted: PipelineDraft
|
||||
): PipelineDraft {
|
||||
const result = { ...current, ...accepted };
|
||||
for (const key of Object.keys(submitted) as Array<keyof PipelineDraft>) {
|
||||
if (key === "id" || key === "currentRevision" || key === "governance") continue;
|
||||
if (JSON.stringify(current[key]) !== JSON.stringify(submitted[key])) {
|
||||
Object.assign(result, { [key]: current[key] });
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
@@ -1,6 +1,11 @@
|
||||
import type { PlatformTranslations } from "@govoplan/core-webui";
|
||||
|
||||
const en = {
|
||||
"A prior save completed after authorization changed. Reload and review the server revision before saving again.": "A prior save completed after authorization changed. Reload and review the server revision before saving again.",
|
||||
"The submitted revision was saved. Newer edits remain unsaved.": "The submitted revision was saved. Newer edits remain unsaved.",
|
||||
"CSV values": "CSV values",
|
||||
"Preserve text (no automatic conversion)": "Preserve text (no automatic conversion)",
|
||||
"Infer types (legacy)": "Infer types (legacy)",
|
||||
"i18n:govoplan-dataflow.dataflow": "Dataflow",
|
||||
"i18n:govoplan-dataflow.library": "Pipeline library",
|
||||
"i18n:govoplan-dataflow.graph": "Graph editor",
|
||||
@@ -56,6 +61,25 @@ const en = {
|
||||
"New pipeline": "New pipeline",
|
||||
"Definition settings": "Definition settings",
|
||||
"Reuse as scoped copy": "Reuse as scoped copy",
|
||||
"Review source update": "Review source update",
|
||||
"Adopt source revision": "Adopt source revision",
|
||||
"Source unavailable": "Source unavailable",
|
||||
"Source update available": "Source update available",
|
||||
"Source definition": "Source definition",
|
||||
"Source revision current": "Source revision current",
|
||||
"Unapplied source update": "Unapplied source update",
|
||||
"Apply the reviewed source update or discard the review reason before leaving.": "Apply the reviewed source update or discard the review reason before leaving.",
|
||||
"The source definition is no longer available.": "The source definition is no longer available.",
|
||||
"This copy already pins the current source revision.": "This copy already pins the current source revision.",
|
||||
"Adopting the update replaces the copy's current graph with the exact reviewed source revision and returns the copy to draft. Existing revisions, run evidence and rebase provenance remain immutable.": "Adopting the update replaces the copy's current graph with the exact reviewed source revision and returns the copy to draft. Existing revisions, run evidence and rebase provenance remain immutable.",
|
||||
"Review reason": "Review reason",
|
||||
"Record what was reviewed and why this source revision is appropriate for the scoped copy.": "Record what was reviewed and why this source revision is appropriate for the scoped copy.",
|
||||
"Reusable input binding": "Reusable input binding",
|
||||
"A reusable definition must mark exactly one typed inline source as the rows supplied by its caller.": "A reusable definition must mark exactly one typed inline source as the rows supplied by its caller.",
|
||||
"Reusable definition": "Reusable definition",
|
||||
"Choose a reusable definition": "Choose a reusable definition",
|
||||
"Template version": "Template version",
|
||||
"Pinned contracts": "Pinned contracts",
|
||||
"Automation triggers": "Automation triggers",
|
||||
"Discard changes": "Discard changes",
|
||||
"Delete pipeline": "Delete pipeline",
|
||||
@@ -81,6 +105,11 @@ const en = {
|
||||
} as const;
|
||||
|
||||
const de: Record<keyof typeof en, string> = {
|
||||
"A prior save completed after authorization changed. Reload and review the server revision before saving again.": "Ein vorheriger Speichervorgang wurde nach einer Berechtigungsänderung abgeschlossen. Vor erneutem Speichern neu laden und die Serverrevision prüfen.",
|
||||
"The submitted revision was saved. Newer edits remain unsaved.": "Die übermittelte Revision wurde gespeichert. Neuere Änderungen sind noch ungespeichert.",
|
||||
"CSV values": "CSV-Werte",
|
||||
"Preserve text (no automatic conversion)": "Text erhalten (keine automatische Umwandlung)",
|
||||
"Infer types (legacy)": "Typen ableiten (bisheriges Verhalten)",
|
||||
"i18n:govoplan-dataflow.dataflow": "Datenfluss",
|
||||
"i18n:govoplan-dataflow.library": "Datenflussbibliothek",
|
||||
"i18n:govoplan-dataflow.graph": "Graph-Editor",
|
||||
@@ -136,6 +165,25 @@ const de: Record<keyof typeof en, string> = {
|
||||
"New pipeline": "Neuer Datenfluss",
|
||||
"Definition settings": "Definitionseinstellungen",
|
||||
"Reuse as scoped copy": "Als eingegrenzte Kopie verwenden",
|
||||
"Review source update": "Aktualisierung der Quelle prüfen",
|
||||
"Adopt source revision": "Quellrevision übernehmen",
|
||||
"Source unavailable": "Quelle nicht verfügbar",
|
||||
"Source update available": "Aktualisierung der Quelle verfügbar",
|
||||
"Source definition": "Quelldefinition",
|
||||
"Source revision current": "Quellrevision aktuell",
|
||||
"Unapplied source update": "Nicht übernommene Quellenaktualisierung",
|
||||
"Apply the reviewed source update or discard the review reason before leaving.": "Übernehmen Sie die geprüfte Quellenaktualisierung oder verwerfen Sie die Prüfbegründung, bevor Sie den Dialog verlassen.",
|
||||
"The source definition is no longer available.": "Die Quelldefinition ist nicht mehr verfügbar.",
|
||||
"This copy already pins the current source revision.": "Diese Kopie ist bereits an die aktuelle Quellrevision gebunden.",
|
||||
"Adopting the update replaces the copy's current graph with the exact reviewed source revision and returns the copy to draft. Existing revisions, run evidence and rebase provenance remain immutable.": "Die Übernahme ersetzt den aktuellen Graphen der Kopie durch die exakt geprüfte Quellrevision und setzt die Kopie auf Entwurf zurück. Bestehende Revisionen, Ausführungsnachweise und die Herkunft der Übernahme bleiben unveränderlich.",
|
||||
"Review reason": "Prüfbegründung",
|
||||
"Record what was reviewed and why this source revision is appropriate for the scoped copy.": "Dokumentieren Sie, was geprüft wurde und warum diese Quellrevision für die eingegrenzte Kopie geeignet ist.",
|
||||
"Reusable input binding": "Wiederverwendbare Eingabebindung",
|
||||
"A reusable definition must mark exactly one typed inline source as the rows supplied by its caller.": "Eine wiederverwendbare Definition muss genau eine typisierte Inline-Quelle als die vom Aufrufer gelieferten Zeilen kennzeichnen.",
|
||||
"Reusable definition": "Wiederverwendbare Definition",
|
||||
"Choose a reusable definition": "Wiederverwendbare Definition auswählen",
|
||||
"Template version": "Vorlagenversion",
|
||||
"Pinned contracts": "Gebundene Verträge",
|
||||
"Automation triggers": "Automatisierungsauslöser",
|
||||
"Discard changes": "Änderungen verwerfen",
|
||||
"Delete pipeline": "Datenfluss löschen",
|
||||
|
||||
+217
-439
@@ -1,69 +1,10 @@
|
||||
.dataflow-page {
|
||||
position: relative;
|
||||
height: calc(100vh - 115px);
|
||||
min-width: 0;
|
||||
min-height: 0;
|
||||
padding: 0;
|
||||
overflow: hidden;
|
||||
color: var(--text);
|
||||
background: var(--bg);
|
||||
}
|
||||
|
||||
.dataflow-page *,
|
||||
.dataflow-page *::before,
|
||||
.dataflow-page *::after {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
.dataflow-shell {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(250px, 300px) minmax(0, 1fr);
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
min-width: 0;
|
||||
min-height: 0;
|
||||
overflow: hidden;
|
||||
border: var(--border-line);
|
||||
background: var(--panel);
|
||||
}
|
||||
|
||||
.dataflow-pipeline-panel,
|
||||
.dataflow-workspace,
|
||||
.dataflow-editor,
|
||||
.dataflow-editor-surface,
|
||||
.dataflow-canvas,
|
||||
.dataflow-pipeline-list-frame {
|
||||
min-width: 0;
|
||||
min-height: 0;
|
||||
}
|
||||
|
||||
.dataflow-pipeline-panel {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
border-right: var(--border-line);
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
|
||||
.dataflow-panel-toolbar,
|
||||
.dataflow-workspace-toolbar,
|
||||
.dataflow-results-toolbar,
|
||||
.dataflow-sql-toolbar,
|
||||
.dataflow-panel-heading {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 10px;
|
||||
flex: 0 0 auto;
|
||||
border-bottom: var(--border-line);
|
||||
background: var(--panel-header);
|
||||
}
|
||||
|
||||
.dataflow-panel-toolbar {
|
||||
min-height: 52px;
|
||||
padding: 8px 10px 8px 14px;
|
||||
}
|
||||
|
||||
.dataflow-toolbar-actions,
|
||||
.dataflow-command-bar,
|
||||
.dataflow-identity-fields {
|
||||
@@ -72,66 +13,11 @@
|
||||
gap: 7px;
|
||||
}
|
||||
|
||||
.dataflow-pipeline-search {
|
||||
padding: 10px;
|
||||
border-bottom: var(--border-line);
|
||||
}
|
||||
|
||||
.dataflow-pipeline-search input {
|
||||
min-height: 34px;
|
||||
padding: 7px 9px;
|
||||
}
|
||||
|
||||
.dataflow-pipeline-list-frame {
|
||||
flex: 1 1 auto;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.dataflow-pipeline-list {
|
||||
height: 100%;
|
||||
overflow: auto;
|
||||
padding: 6px;
|
||||
}
|
||||
|
||||
.dataflow-pipeline-list > button {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(0, 1fr) auto;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
width: 100%;
|
||||
min-height: 56px;
|
||||
border: 0;
|
||||
border-radius: var(--radius-sm);
|
||||
background: transparent;
|
||||
color: var(--text);
|
||||
cursor: pointer;
|
||||
padding: 8px 9px;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.dataflow-pipeline-list > button:hover,
|
||||
.dataflow-pipeline-list > button:focus-visible {
|
||||
background: var(--primary-soft);
|
||||
outline: none;
|
||||
}
|
||||
|
||||
.dataflow-pipeline-list > button.is-selected {
|
||||
background: var(--primary-soft-strong);
|
||||
box-shadow: inset 3px 0 0 var(--accent);
|
||||
}
|
||||
|
||||
.dataflow-pipeline-list > button > span:first-child {
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.dataflow-pipeline-list strong,
|
||||
.dataflow-pipeline-list small {
|
||||
display: block;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.dataflow-definition-dialog {
|
||||
width: min(620px, calc(100vw - 32px));
|
||||
}
|
||||
@@ -150,13 +36,6 @@
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.dataflow-definition-toggles {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
gap: 10px 18px;
|
||||
}
|
||||
|
||||
.dataflow-provenance,
|
||||
.dataflow-trigger-security {
|
||||
display: grid;
|
||||
gap: 6px;
|
||||
@@ -241,8 +120,7 @@
|
||||
}
|
||||
|
||||
@media (max-width: 760px) {
|
||||
.dataflow-triggers-layout,
|
||||
.dataflow-definition-toggles {
|
||||
.dataflow-triggers-layout {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
@@ -254,49 +132,12 @@
|
||||
}
|
||||
}
|
||||
|
||||
.dataflow-pipeline-list strong {
|
||||
color: var(--text-strong);
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.dataflow-pipeline-list small {
|
||||
margin-top: 4px;
|
||||
color: var(--muted);
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.dataflow-pipeline-list .status-badge {
|
||||
max-width: 74px;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
}
|
||||
|
||||
.dataflow-pipeline-empty,
|
||||
.dataflow-inspector-empty,
|
||||
.dataflow-results-empty {
|
||||
display: grid;
|
||||
place-items: center;
|
||||
min-height: 100px;
|
||||
color: var(--muted);
|
||||
font-size: 13px;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.dataflow-workspace {
|
||||
position: relative;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
background: var(--bg);
|
||||
}
|
||||
|
||||
.dataflow-action-guidance {
|
||||
padding: 0 10px 8px;
|
||||
}
|
||||
|
||||
.dataflow-workspace-toolbar {
|
||||
min-height: 58px;
|
||||
padding: 8px 10px;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.dataflow-identity-fields {
|
||||
@@ -368,7 +209,6 @@
|
||||
outline-offset: -2px;
|
||||
}
|
||||
|
||||
.dataflow-palette,
|
||||
.dataflow-inspector {
|
||||
min-width: 0;
|
||||
min-height: 0;
|
||||
@@ -376,19 +216,10 @@
|
||||
background: var(--panel-soft);
|
||||
}
|
||||
|
||||
.dataflow-palette {
|
||||
border-right: var(--border-line);
|
||||
}
|
||||
|
||||
.dataflow-inspector {
|
||||
border-left: var(--border-line);
|
||||
}
|
||||
|
||||
.dataflow-panel-heading {
|
||||
min-height: 44px;
|
||||
padding: 8px 10px 8px 12px;
|
||||
}
|
||||
|
||||
.dataflow-panel-heading > span {
|
||||
min-width: 0;
|
||||
}
|
||||
@@ -413,127 +244,6 @@
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.dataflow-palette-items {
|
||||
display: grid;
|
||||
grid-auto-rows: max-content;
|
||||
align-content: start;
|
||||
gap: 4px;
|
||||
height: calc(100% - 44px);
|
||||
padding: 8px;
|
||||
overflow-x: hidden;
|
||||
overflow-y: auto;
|
||||
scrollbar-gutter: stable;
|
||||
}
|
||||
|
||||
.dataflow-palette-group {
|
||||
display: grid;
|
||||
grid-auto-rows: max-content;
|
||||
align-content: start;
|
||||
gap: 2px;
|
||||
}
|
||||
|
||||
.dataflow-palette-group + .dataflow-palette-group {
|
||||
margin-top: 6px;
|
||||
padding-top: 8px;
|
||||
border-top: var(--border-line);
|
||||
}
|
||||
|
||||
.dataflow-palette-group h3 {
|
||||
margin: 0;
|
||||
color: var(--muted);
|
||||
font-size: 10px;
|
||||
font-weight: 700;
|
||||
letter-spacing: 0;
|
||||
padding: 3px 8px;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.dataflow-palette-items button {
|
||||
display: grid;
|
||||
grid-template-columns: 18px minmax(0, 1fr) 14px;
|
||||
align-items: center;
|
||||
gap: 7px;
|
||||
min-height: 38px;
|
||||
border: 0;
|
||||
border-radius: var(--radius-sm);
|
||||
background: transparent;
|
||||
color: var(--text);
|
||||
cursor: grab;
|
||||
font: inherit;
|
||||
font-size: 12px;
|
||||
padding: 7px 8px;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.dataflow-palette-items button:hover:not(:disabled),
|
||||
.dataflow-palette-items button:focus-visible:not(:disabled) {
|
||||
background: var(--primary-soft);
|
||||
color: var(--text-strong);
|
||||
outline: none;
|
||||
}
|
||||
|
||||
.dataflow-palette-items button:active:not(:disabled) {
|
||||
cursor: grabbing;
|
||||
}
|
||||
|
||||
.dataflow-palette-items button:disabled {
|
||||
cursor: default;
|
||||
opacity: .42;
|
||||
}
|
||||
|
||||
.dataflow-palette-add {
|
||||
color: var(--accent);
|
||||
}
|
||||
|
||||
.dataflow-editor-surface {
|
||||
position: relative;
|
||||
overflow: hidden;
|
||||
background: var(--bg);
|
||||
}
|
||||
|
||||
.dataflow-canvas {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
}
|
||||
|
||||
.dataflow-canvas .react-flow {
|
||||
background: var(--bg);
|
||||
}
|
||||
|
||||
.dataflow-canvas .react-flow__background {
|
||||
color: var(--line-dark);
|
||||
}
|
||||
|
||||
.dataflow-canvas .react-flow__controls,
|
||||
.dataflow-canvas .react-flow__minimap {
|
||||
overflow: hidden;
|
||||
border: var(--border-line);
|
||||
border-radius: var(--radius-sm);
|
||||
background: var(--panel);
|
||||
box-shadow: var(--shadow-xs);
|
||||
}
|
||||
|
||||
.dataflow-canvas .react-flow__controls-button {
|
||||
border-bottom: var(--border-line);
|
||||
background: var(--panel);
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.dataflow-canvas .react-flow__minimap-mask {
|
||||
fill: color-mix(in srgb, var(--bg) 76%, transparent);
|
||||
}
|
||||
|
||||
.dataflow-canvas .react-flow__edge-path {
|
||||
stroke: var(--line-dark);
|
||||
stroke-width: 2;
|
||||
}
|
||||
|
||||
.dataflow-canvas .react-flow__edge.selected .react-flow__edge-path,
|
||||
.dataflow-canvas .react-flow__edge:hover .react-flow__edge-path {
|
||||
stroke: var(--accent);
|
||||
stroke-width: 3;
|
||||
}
|
||||
|
||||
.dataflow-canvas .react-flow__edge.dataflow-edge-proximity .react-flow__edge-path {
|
||||
stroke: var(--accent);
|
||||
stroke-width: 3;
|
||||
@@ -545,15 +255,6 @@
|
||||
stroke-width: 3;
|
||||
}
|
||||
|
||||
.dataflow-canvas-empty {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
color: var(--muted);
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
.dataflow-node {
|
||||
position: relative;
|
||||
display: flex;
|
||||
@@ -562,8 +263,8 @@
|
||||
width: 180px;
|
||||
min-height: 54px;
|
||||
border: 1px solid var(--line-dark);
|
||||
border-left: 4px solid #3d6f9e;
|
||||
border-radius: 6px;
|
||||
border-left: 4px solid var(--data-category-blue);
|
||||
border-radius: var(--radius-compact);
|
||||
background: var(--panel);
|
||||
box-shadow: var(--shadow-xs);
|
||||
padding: 8px 10px;
|
||||
@@ -571,34 +272,34 @@
|
||||
|
||||
.dataflow-node-source-inline,
|
||||
.dataflow-node-source-reference {
|
||||
border-left-color: #2f7d6d;
|
||||
border-left-color: var(--data-category-green);
|
||||
}
|
||||
|
||||
.dataflow-node-filter {
|
||||
border-left-color: #b7791f;
|
||||
border-left-color: var(--data-category-amber);
|
||||
}
|
||||
|
||||
.dataflow-node-combine-union,
|
||||
.dataflow-node-combine-join {
|
||||
min-height: 64px;
|
||||
border-left-color: #2f7d6d;
|
||||
border-left-color: var(--data-category-green);
|
||||
}
|
||||
|
||||
.dataflow-node-distinct {
|
||||
border-left-color: #b7791f;
|
||||
border-left-color: var(--data-category-amber);
|
||||
}
|
||||
|
||||
.dataflow-node-derive,
|
||||
.dataflow-node-aggregate {
|
||||
border-left-color: #76569b;
|
||||
border-left-color: var(--data-category-purple);
|
||||
}
|
||||
|
||||
.dataflow-node-sort {
|
||||
border-left-color: #3d6f9e;
|
||||
border-left-color: var(--data-category-blue);
|
||||
}
|
||||
|
||||
.dataflow-node-output {
|
||||
border-left-color: #9d4e63;
|
||||
border-left-color: var(--data-category-rose);
|
||||
}
|
||||
|
||||
.dataflow-node.is-selected {
|
||||
@@ -610,17 +311,6 @@
|
||||
border-color: var(--danger-text);
|
||||
}
|
||||
|
||||
.dataflow-node-icon {
|
||||
display: grid;
|
||||
width: 28px;
|
||||
height: 28px;
|
||||
flex: 0 0 28px;
|
||||
place-items: center;
|
||||
border-radius: 5px;
|
||||
background: var(--panel-soft);
|
||||
color: var(--text-strong);
|
||||
}
|
||||
|
||||
.dataflow-node-copy {
|
||||
min-width: 0;
|
||||
flex: 1 1 auto;
|
||||
@@ -645,32 +335,6 @@
|
||||
font-size: 10px;
|
||||
}
|
||||
|
||||
.dataflow-node-count {
|
||||
min-width: 22px;
|
||||
border-radius: 999px;
|
||||
background: var(--primary-soft);
|
||||
color: var(--text-strong);
|
||||
font-size: 10px;
|
||||
font-variant-numeric: tabular-nums;
|
||||
padding: 3px 5px;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.dataflow-node-handle {
|
||||
width: 13px;
|
||||
height: 13px;
|
||||
border: 3px solid var(--panel);
|
||||
background: var(--line-dark);
|
||||
}
|
||||
|
||||
.dataflow-node-handle:hover,
|
||||
.dataflow-node-handle.connectingto,
|
||||
.dataflow-node-handle.valid {
|
||||
width: 17px;
|
||||
height: 17px;
|
||||
background: var(--accent);
|
||||
}
|
||||
|
||||
.dataflow-node-port-label {
|
||||
position: absolute;
|
||||
left: 8px;
|
||||
@@ -685,7 +349,7 @@
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.dataflow-node-combine-join .dataflow-node-icon {
|
||||
.dataflow-node-combine-join .definition-node-icon {
|
||||
margin-left: 31px;
|
||||
}
|
||||
|
||||
@@ -736,6 +400,210 @@
|
||||
width: min(720px, calc(100vw - 32px));
|
||||
}
|
||||
|
||||
.dataflow-decision-dialog {
|
||||
width: min(1080px, calc(100vw - 32px));
|
||||
height: min(760px, calc(100vh - 48px));
|
||||
}
|
||||
|
||||
.dataflow-decision-dialog-body {
|
||||
overflow: hidden;
|
||||
padding: 14px;
|
||||
}
|
||||
|
||||
.dataflow-decision-shell {
|
||||
display: flex;
|
||||
height: 100%;
|
||||
min-height: 0;
|
||||
flex-direction: column;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.dataflow-decision-set-toolbar {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(190px, 0.8fr) minmax(300px, 1.2fr) auto;
|
||||
align-items: end;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.dataflow-decision-set-toolbar select,
|
||||
.dataflow-decision-set-toolbar input,
|
||||
.dataflow-decision-editor textarea {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.dataflow-decision-create,
|
||||
.dataflow-decision-counts,
|
||||
.dataflow-results-actions,
|
||||
.dataflow-decision-editor-actions {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.dataflow-decision-create input {
|
||||
min-width: 0;
|
||||
flex: 1 1 auto;
|
||||
}
|
||||
|
||||
.dataflow-decision-create .btn {
|
||||
flex: 0 0 auto;
|
||||
}
|
||||
|
||||
.dataflow-decision-counts {
|
||||
min-height: 34px;
|
||||
color: var(--muted);
|
||||
font-size: 11px;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.dataflow-decision-counts strong {
|
||||
color: var(--text-strong);
|
||||
}
|
||||
|
||||
.dataflow-decision-counts .is-warning strong {
|
||||
color: var(--warning-deep);
|
||||
}
|
||||
|
||||
.dataflow-decision-layout {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(420px, 1.1fr) minmax(340px, 0.9fr);
|
||||
min-height: 0;
|
||||
flex: 1 1 auto;
|
||||
overflow: hidden;
|
||||
border: var(--border-line);
|
||||
border-radius: var(--radius-sm);
|
||||
}
|
||||
|
||||
.dataflow-decision-rows,
|
||||
.dataflow-decision-editor {
|
||||
min-width: 0;
|
||||
min-height: 0;
|
||||
overflow: auto;
|
||||
}
|
||||
|
||||
.dataflow-decision-rows {
|
||||
border-right: var(--border-line);
|
||||
}
|
||||
|
||||
.dataflow-decision-row-heading,
|
||||
.dataflow-decision-rows > button {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(180px, 1fr) minmax(90px, 0.45fr) 90px;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
width: 100%;
|
||||
min-height: 42px;
|
||||
border: 0;
|
||||
border-bottom: var(--border-line);
|
||||
padding: 7px 10px;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.dataflow-decision-row-heading {
|
||||
position: sticky;
|
||||
z-index: 2;
|
||||
top: 0;
|
||||
background: var(--panel-header);
|
||||
color: var(--muted);
|
||||
font-size: 10px;
|
||||
font-weight: 700;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.dataflow-decision-rows > button {
|
||||
background: transparent;
|
||||
color: var(--text);
|
||||
cursor: pointer;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.dataflow-decision-rows > button:hover,
|
||||
.dataflow-decision-rows > button.is-selected {
|
||||
background: var(--primary-soft);
|
||||
}
|
||||
|
||||
.dataflow-decision-rows > button.is-selected {
|
||||
box-shadow: inset 3px 0 0 var(--accent);
|
||||
}
|
||||
|
||||
.dataflow-decision-rows > button > span {
|
||||
min-width: 0;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.dataflow-decision-state {
|
||||
color: var(--muted);
|
||||
font-size: 10px;
|
||||
font-weight: 700;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.dataflow-decision-state.is-accept,
|
||||
.dataflow-decision-state.is-correct {
|
||||
color: var(--success-text);
|
||||
}
|
||||
|
||||
.dataflow-decision-state.is-reject,
|
||||
.dataflow-decision-state.is-stale {
|
||||
color: var(--danger-text);
|
||||
}
|
||||
|
||||
.dataflow-decision-state.is-defer {
|
||||
color: var(--warning-deep);
|
||||
}
|
||||
|
||||
.dataflow-decision-editor {
|
||||
display: grid;
|
||||
grid-auto-rows: max-content;
|
||||
align-content: start;
|
||||
gap: 14px;
|
||||
padding: 14px;
|
||||
}
|
||||
|
||||
.dataflow-decision-editor-heading {
|
||||
display: flex;
|
||||
align-items: start;
|
||||
justify-content: space-between;
|
||||
gap: 12px;
|
||||
padding-bottom: 10px;
|
||||
border-bottom: var(--border-line);
|
||||
}
|
||||
|
||||
.dataflow-decision-editor-heading > div {
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.dataflow-decision-editor-heading strong,
|
||||
.dataflow-decision-editor-heading small {
|
||||
display: block;
|
||||
}
|
||||
|
||||
.dataflow-decision-editor-heading small {
|
||||
margin-top: 4px;
|
||||
overflow: hidden;
|
||||
color: var(--muted);
|
||||
font-family: ui-monospace, SFMono-Regular, Consolas, monospace;
|
||||
font-size: 9px;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.dataflow-decision-editor .dataflow-json-editor {
|
||||
min-height: 140px;
|
||||
}
|
||||
|
||||
.dataflow-decision-editor-actions {
|
||||
justify-content: flex-end;
|
||||
}
|
||||
|
||||
.dataflow-decision-history-summary {
|
||||
margin-right: auto;
|
||||
color: var(--muted);
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.dataflow-run-dialog {
|
||||
width: min(860px, calc(100vw - 32px));
|
||||
}
|
||||
@@ -873,8 +741,6 @@
|
||||
}
|
||||
|
||||
.dataflow-sql-toolbar {
|
||||
min-height: 44px;
|
||||
padding: 7px 10px 7px 14px;
|
||||
color: var(--text-strong);
|
||||
font-size: 12px;
|
||||
font-weight: 700;
|
||||
@@ -911,8 +777,6 @@
|
||||
}
|
||||
|
||||
.dataflow-results-toolbar {
|
||||
min-height: 42px;
|
||||
padding: 5px 9px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
@@ -1051,44 +915,7 @@
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.dataflow-workspace-empty {
|
||||
display: grid;
|
||||
place-content: center;
|
||||
justify-items: center;
|
||||
gap: 12px;
|
||||
height: 100%;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.dataflow-workspace-empty strong {
|
||||
color: var(--text-strong);
|
||||
}
|
||||
|
||||
.dataflow-workspace-empty .btn {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
}
|
||||
|
||||
.dataflow-working-indicator {
|
||||
position: absolute;
|
||||
z-index: 20;
|
||||
right: 12px;
|
||||
bottom: 10px;
|
||||
border: var(--border-line-dark);
|
||||
border-radius: var(--radius-sm);
|
||||
background: var(--panel);
|
||||
box-shadow: var(--shadow-popover);
|
||||
color: var(--text-strong);
|
||||
font-size: 12px;
|
||||
padding: 7px 10px;
|
||||
}
|
||||
|
||||
@media (max-width: 1180px) {
|
||||
.dataflow-shell {
|
||||
grid-template-columns: 240px minmax(0, 1fr);
|
||||
}
|
||||
|
||||
@media (max-width: 1280px) {
|
||||
.dataflow-workspace-toolbar {
|
||||
align-items: stretch;
|
||||
flex-direction: column;
|
||||
@@ -1108,10 +935,6 @@
|
||||
}
|
||||
|
||||
@media (max-width: 900px) {
|
||||
.dataflow-shell {
|
||||
grid-template-columns: 210px minmax(0, 1fr);
|
||||
}
|
||||
|
||||
.dataflow-editor {
|
||||
grid-template-columns: 135px minmax(0, 1fr);
|
||||
grid-template-rows: minmax(0, 1fr) minmax(170px, 32%);
|
||||
@@ -1144,16 +967,6 @@
|
||||
height: calc(100dvh - 115px);
|
||||
}
|
||||
|
||||
.dataflow-shell {
|
||||
grid-template-columns: 1fr;
|
||||
grid-template-rows: minmax(150px, 24%) minmax(0, 1fr);
|
||||
}
|
||||
|
||||
.dataflow-pipeline-panel {
|
||||
border-right: 0;
|
||||
border-bottom: var(--border-line);
|
||||
}
|
||||
|
||||
.dataflow-workspace-toolbar {
|
||||
overflow-x: auto;
|
||||
align-items: flex-start;
|
||||
@@ -1179,41 +992,6 @@
|
||||
grid-template-rows: auto minmax(0, 1fr) minmax(160px, 30%);
|
||||
}
|
||||
|
||||
.dataflow-palette {
|
||||
border-right: 0;
|
||||
border-bottom: var(--border-line);
|
||||
}
|
||||
|
||||
.dataflow-palette .dataflow-panel-heading {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.dataflow-palette-items {
|
||||
display: flex;
|
||||
overflow-x: auto;
|
||||
}
|
||||
|
||||
.dataflow-palette-group {
|
||||
display: flex;
|
||||
flex: 0 0 auto;
|
||||
}
|
||||
|
||||
.dataflow-palette-group + .dataflow-palette-group {
|
||||
margin-top: 0;
|
||||
padding-top: 0;
|
||||
padding-left: 6px;
|
||||
border-top: 0;
|
||||
border-left: var(--border-line);
|
||||
}
|
||||
|
||||
.dataflow-palette-group h3 {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.dataflow-palette-items button {
|
||||
min-width: 128px;
|
||||
}
|
||||
|
||||
.dataflow-inspector-fields {
|
||||
grid-template-columns: 1fr 1fr;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user