[Feature] Add scoped Dataflow templates, inheritance, derivation, and reuse provenance #12

New Issue

2026-07-28T14:02:58+02:00

zemion commented

2026-07-28 14:02:58 +02:00

Parent: #1

Make Dataflow definitions reusable across governance scopes without losing ownership, immutability, or explainability.

Acceptance criteria:

definitions carry system/tenant/group/user scope, flow/template kind, inheritance, run, and reuse grants
inherited definitions are visible read-only and every disabled edit/run/reuse action explains the source policy
templates are never runnable; they can create a local definition pinned to an immutable source revision
reusable complete flows can be derived or referenced only when Policy permits it
derived definitions retain source definition, revision, hash, actor, policy decision, and update-available provenance
parent changes never silently mutate an active child; rebase/update is explicit and reviewable
system and tenant libraries are usable from the graphical editor and API
cycles across nested flow references and incompatible input/output contracts are rejected

Parent: https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/1 Make Dataflow definitions reusable across governance scopes without losing ownership, immutability, or explainability. Acceptance criteria: - definitions carry system/tenant/group/user scope, flow/template kind, inheritance, run, and reuse grants - inherited definitions are visible read-only and every disabled edit/run/reuse action explains the source policy - templates are never runnable; they can create a local definition pinned to an immutable source revision - reusable complete flows can be derived or referenced only when Policy permits it - derived definitions retain source definition, revision, hash, actor, policy decision, and update-available provenance - parent changes never silently mutate an active child; rebase/update is explicit and reviewable - system and tenant libraries are usable from the graphical editor and API - cycles across nested flow references and incompatible input/output contracts are rejected

zemion added the area/api area/db area/governance area/webui codex/ready module/dataflow module/policy

labels 2026-07-28 14:02:59 +02:00

zemion commented

2026-07-28 15:06:40 +02:00

Governed reusable definitions implemented in govoplan-core@26ae034, govoplan-policy@d6e09fb, and govoplan-dataflow@08ddc8d.

Completed:

system/tenant/group/user scope and flow/template kind
inheritance, run, reuse, and automation limits with explainable action decisions
inherited read-only UI and disabled-action reasons
templates cannot execute or receive active automation
pinned-copy derivation records source ID/revision/hash/scope, actor, Policy decision, and effective limits
ancestor ceilings survive edits and multiple derivation generations
system and tenant definitions appear in the API/editor library
parent edits never silently mutate a child

Remaining before closure:

update-available detection and explicit reviewed rebase
reference-based nested flow composition, typed input/output contracts, and cycle rejection

Verification: 35 Dataflow tests plus WebUI and migration checks pass.

Governed reusable definitions implemented in govoplan-core@26ae034, govoplan-policy@d6e09fb, and govoplan-dataflow@08ddc8d. Completed: - system/tenant/group/user scope and flow/template kind - inheritance, run, reuse, and automation limits with explainable action decisions - inherited read-only UI and disabled-action reasons - templates cannot execute or receive active automation - pinned-copy derivation records source ID/revision/hash/scope, actor, Policy decision, and effective limits - ancestor ceilings survive edits and multiple derivation generations - system and tenant definitions appear in the API/editor library - parent edits never silently mutate a child Remaining before closure: - update-available detection and explicit reviewed rebase - reference-based nested flow composition, typed input/output contracts, and cycle rejection Verification: 35 Dataflow tests plus WebUI and migration checks pass.

zemion referenced this issue

2026-07-29 14:27:58 +02:00

[Feature] Introduce typed Dataflow IR and an engine-neutral Arrow execution boundary #15

zemion commented

2026-07-29 14:28:32 +02:00

Parameterized template refinement

Parameterized composite templates belong in this issue rather than a parallel template system. A template should expose typed input/output ports and a bounded parameter schema while keeping its internal graph/version immutable for a given revision. Derivations pin the template revision, parameter values/defaults, semantic hash, actor, Policy decision, and transitive limits. Nested templates require explicit compatibility checks and cycle rejection. Templates remain non-runnable.

The typed IR and composite-node contracts needed for this are tracked by #15.

## Parameterized template refinement Parameterized composite templates belong in this issue rather than a parallel template system. A template should expose typed input/output ports and a bounded parameter schema while keeping its internal graph/version immutable for a given revision. Derivations pin the template revision, parameter values/defaults, semantic hash, actor, Policy decision, and transitive limits. Nested templates require explicit compatibility checks and cycle rejection. Templates remain non-runnable. The typed IR and composite-node contracts needed for this are tracked by [#15](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/15).

zemion commented

2026-07-29 15:52:48 +02:00

Codex State: progress

Summary

Added a reusable subflow operator with pinned template reference/version, bounded structured parameters, one explicit input binding, nested graph validation, and execution-depth protection.
Added typed expression, conversion, replacement, quality, and reconciliation nodes with schema propagation and editor controls.

Verification

Parameterized subflow and operator tests passed

Next / Blocked

Persisted immutable template revision resolution, full governance-policy enforcement, update/rebase provenance, and system/tenant editor libraries remain.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Added a reusable subflow operator with pinned template reference/version, bounded structured parameters, one explicit input binding, nested graph validation, and execution-depth protection. - Added typed expression, conversion, replacement, quality, and reconciliation nodes with schema propagation and editor controls. ### Verification - `Parameterized subflow and operator tests passed` ### Next / Blocked - Persisted immutable template revision resolution, full governance-policy enforcement, update/rebase provenance, and system/tenant editor libraries remain. Suggested status label: `status/in-progress`

zemion referenced this issue

2026-08-03 00:05:07 +02:00

[Epic] Deliver a governed Dataflow manager with graphical and SQL editing #1

zemion commented

2026-08-21 18:46:52 +02:00

Implemented and released in c1111c6 (v0.1.20).

Acceptance evidence:

Reusable definitions retain scope, kind, grants, inheritance, derivation, and immutable source provenance.
Derived pipelines detect source revision/hash drift without silently changing the child.
The reviewed rebase operation requires the exact current source revision/hash plus a meaningful reason, creates a new immutable child revision, preserves prior graph/provenance history, narrows governance ceilings, and returns the child to draft for review/reactivation.
Reusable subflows are selected from the governed pipeline library and resolved server-side from canonical pipeline:<id> references plus immutable revisions. Caller-pasted graphs are ignored.
Every reuse is authorized through the current Policy decision and pins the source hash, scope/kind, policy proof, nested graph, and closed typed input/output contracts.
Invalid or untyped bindings, incompatible caller schemas, self-reference, and nested/cross-revision cycles are rejected before persistence or preview.
The WebUI provides controlled source/revision selection, reusable input bindings, source-update review, dirty-state guards, exact help contexts, and German translations.
README and manifest documentation describe administrator/user workflows, permissions, limitations, and operational consequences.

Verification:

Dataflow: 95 tests passed plus 29 subtests; Ruff, WebUI typecheck, and structure checks passed.
Workspace: manifest shapes 68/68; 68 modules, 152 providers, and 94 requirements passed contract checks; dependency/inventory/i18n audits passed with 1,587/1,587 help contexts and no duplicate IDs, missing declarations, stale endpoints, or missing German keys.
tools/checks/check-focused.sh passed end to end, including all backend/module suites, 59 WebUI module permutations, the full-product bundle budget, and 8/8 Playwright conformance tests.

The workspace meta package was synchronized in GovOPlaN/govoplan@60e04a3.

Implemented and released in c1111c6 (`v0.1.20`). Acceptance evidence: - Reusable definitions retain scope, kind, grants, inheritance, derivation, and immutable source provenance. - Derived pipelines detect source revision/hash drift without silently changing the child. - The reviewed rebase operation requires the exact current source revision/hash plus a meaningful reason, creates a new immutable child revision, preserves prior graph/provenance history, narrows governance ceilings, and returns the child to draft for review/reactivation. - Reusable subflows are selected from the governed pipeline library and resolved server-side from canonical `pipeline:<id>` references plus immutable revisions. Caller-pasted graphs are ignored. - Every reuse is authorized through the current Policy decision and pins the source hash, scope/kind, policy proof, nested graph, and closed typed input/output contracts. - Invalid or untyped bindings, incompatible caller schemas, self-reference, and nested/cross-revision cycles are rejected before persistence or preview. - The WebUI provides controlled source/revision selection, reusable input bindings, source-update review, dirty-state guards, exact help contexts, and German translations. - README and manifest documentation describe administrator/user workflows, permissions, limitations, and operational consequences. Verification: - Dataflow: 95 tests passed plus 29 subtests; Ruff, WebUI typecheck, and structure checks passed. - Workspace: manifest shapes 68/68; 68 modules, 152 providers, and 94 requirements passed contract checks; dependency/inventory/i18n audits passed with 1,587/1,587 help contexts and no duplicate IDs, missing declarations, stale endpoints, or missing German keys. - `tools/checks/check-focused.sh` passed end to end, including all backend/module suites, 59 WebUI module permutations, the full-product bundle budget, and 8/8 Playwright conformance tests. The workspace meta package was synchronized in GovOPlaN/govoplan@60e04a3.

zemion closed this issue

2026-08-21 18:46:53 +02:00

zemion referenced this issue from GovOPlaN/govoplan

2026-09-08 05:27:54 +02:00

[Roadmap] Deliver governed dataflows, search, and established-tool integration #19

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-dataflow#12