[Feature] Add governed manual, scheduled, event, API, and chained Dataflow triggers #11

New Issue

2026-07-28T14:02:56+02:00

zemion commented

2026-07-28 14:02:56 +02:00

Parent: #10

Persist and dispatch governed Dataflow trigger definitions. Invocation provenance must distinguish interactive user, API, scheduled, event, Workflow, dependency/upstream, retry/replay, and backfill runs.

Acceptance criteria:

manual/API run origin is persisted on every run
one-time and interval schedules support timezone, next-fire calculation, catch-up policy, enable/disable, and pinned immutable pipeline revision
event triggers accept exact allowlisted event types and constrained equality filters with idempotent event IDs
Workflow/dependency invocation uses the versioned run capability and records correlation/causation references
triggers cannot target templates and automated execution requires an active complete flow
execution resolves current authorization through Access, applies Policy, and fails closed if either required provider is unavailable
overlap, concurrency, quota, retry/quarantine, cancellation, notifications, and audit evidence are explicit
schedule dispatch runs in a worker; arbitrary SQL and large execution never run in the API process

Parent: https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/10 Persist and dispatch governed Dataflow trigger definitions. Invocation provenance must distinguish interactive user, API, scheduled, event, Workflow, dependency/upstream, retry/replay, and backfill runs. Acceptance criteria: - manual/API run origin is persisted on every run - one-time and interval schedules support timezone, next-fire calculation, catch-up policy, enable/disable, and pinned immutable pipeline revision - event triggers accept exact allowlisted event types and constrained equality filters with idempotent event IDs - Workflow/dependency invocation uses the versioned run capability and records correlation/causation references - triggers cannot target templates and automated execution requires an active complete flow - execution resolves current authorization through Access, applies Policy, and fails closed if either required provider is unavailable - overlap, concurrency, quota, retry/quarantine, cancellation, notifications, and audit evidence are explicit - schedule dispatch runs in a worker; arbitrary SQL and large execution never run in the API process

zemion added the area/api area/db area/security area/webui codex/ready module/dataflow module/workflow

labels 2026-07-28 14:02:58 +02:00

zemion commented

2026-07-28 15:06:40 +02:00

Large first slice implemented in govoplan-core@26ae034, govoplan-access@fdfcfbb, govoplan-policy@d6e09fb, govoplan-datasources@6294362, and govoplan-dataflow@08ddc8d.

Completed:

persisted manual, API, and administrative backfill run provenance
durable one-time, interval, and event triggers pinned to immutable revisions
timezone-aware next-fire calculation without unbounded anchor loops
coalesce/skip catch-up behavior with durable skipped-delivery evidence
enable/disable, optimistic trigger revisions, explicit concurrency ceiling, idempotent event IDs, and exact scalar filters
Celery beat/worker dispatch capability and dedicated dataflow queue
current Access authorization plus Policy recheck for every delivery
least-privilege source/publication grants and template/inactive-flow denial
correlation/causation, audit events, delivery status/error, and trigger UI

Remaining before closure:

durable transactional event ingress via govoplan-core#270
actual Workflow/dependency invocation path
asynchronous large-run queue, retries/quarantine, quotas, cancellation, progress, and notifications under govoplan-dataflow#10
manual replay/recovery controls

Verification: 35 Dataflow tests, WebUI typecheck/structure checks, migrations, and consolidated focused suite pass.

Large first slice implemented in govoplan-core@26ae034, govoplan-access@fdfcfbb, govoplan-policy@d6e09fb, govoplan-datasources@6294362, and govoplan-dataflow@08ddc8d. Completed: - persisted manual, API, and administrative backfill run provenance - durable one-time, interval, and event triggers pinned to immutable revisions - timezone-aware next-fire calculation without unbounded anchor loops - coalesce/skip catch-up behavior with durable skipped-delivery evidence - enable/disable, optimistic trigger revisions, explicit concurrency ceiling, idempotent event IDs, and exact scalar filters - Celery beat/worker dispatch capability and dedicated dataflow queue - current Access authorization plus Policy recheck for every delivery - least-privilege source/publication grants and template/inactive-flow denial - correlation/causation, audit events, delivery status/error, and trigger UI Remaining before closure: - durable transactional event ingress via govoplan-core#270 - actual Workflow/dependency invocation path - asynchronous large-run queue, retries/quarantine, quotas, cancellation, progress, and notifications under govoplan-dataflow#10 - manual replay/recovery controls Verification: 35 Dataflow tests, WebUI typecheck/structure checks, migrations, and consolidated focused suite pass.

zemion referenced this issue

2026-08-03 00:05:07 +02:00

[Epic] Deliver a governed Dataflow manager with graphical and SQL editing #1

zemion commented

2026-08-21 18:06:42 +02:00

Current-state verification shows this issue is complete. The July progress note predates the production execution and Workflow lifecycle releases.

Acceptance mapping:

Every run persists a typed invocation kind and immutable request evidence; interactive, API, schedule, event, Workflow, dependency, retry/replay, and backfill origins are represented by the Core automation contract.
One-time and interval triggers persist timezone, next-fire state, skip/coalesce catch-up policy, enablement, concurrency, and a pinned pipeline revision.
Event delivery uses exact event types, bounded scalar equality filters, durable event IDs, correlation/causation, and idempotent delivery rows. Restricted events fail closed unless they arrive through Core's durable bridge.
Workflow and dependency callers use dataflow.runLifecycle; the Workflow instance-service suite covers wait, retry, review, approval, output references, and linked cancellation.
Templates and inactive/incomplete flows cannot execute or receive automation.
Every delivery rebuilds current authority through Access and re-applies Policy. A missing Access provider or missing Policy automation decision blocks delivery.
The worker layer provides tenant queue/concurrency quotas, per-trigger overlap ceilings, bounded exponential retry, cancellation checkpoints, notifications, audit/recovery evidence, progress, retention, and metrics. Unsafe uncertain external effects enter the explicit outcome_unknown quarantine and cannot retry blindly.
API calls only enqueue saved runs; scheduled dispatch and bounded/isolated execution run in workers.

Relevant delivery points include 5af0293, a86220d, Core fbea74a, and the existing Workflow lifecycle implementation. Verification includes all 92 Dataflow tests, focused Workflow lifecycle tests, TypeScript, manifest/contracts/dependency checks, strict platform inventory, all 59 WebUI permutations, eight browser-conformance scenarios, and the complete focused workspace gate.

Current-state verification shows this issue is complete. The July progress note predates the production execution and Workflow lifecycle releases. Acceptance mapping: - Every run persists a typed invocation kind and immutable request evidence; interactive, API, schedule, event, Workflow, dependency, retry/replay, and backfill origins are represented by the Core automation contract. - One-time and interval triggers persist timezone, next-fire state, skip/coalesce catch-up policy, enablement, concurrency, and a pinned pipeline revision. - Event delivery uses exact event types, bounded scalar equality filters, durable event IDs, correlation/causation, and idempotent delivery rows. Restricted events fail closed unless they arrive through Core's durable bridge. - Workflow and dependency callers use `dataflow.runLifecycle`; the Workflow instance-service suite covers wait, retry, review, approval, output references, and linked cancellation. - Templates and inactive/incomplete flows cannot execute or receive automation. - Every delivery rebuilds current authority through Access and re-applies Policy. A missing Access provider or missing Policy automation decision blocks delivery. - The worker layer provides tenant queue/concurrency quotas, per-trigger overlap ceilings, bounded exponential retry, cancellation checkpoints, notifications, audit/recovery evidence, progress, retention, and metrics. Unsafe uncertain external effects enter the explicit `outcome_unknown` quarantine and cannot retry blindly. - API calls only enqueue saved runs; scheduled dispatch and bounded/isolated execution run in workers. Relevant delivery points include `5af0293`, `a86220d`, Core `fbea74a`, and the existing Workflow lifecycle implementation. Verification includes all 92 Dataflow tests, focused Workflow lifecycle tests, TypeScript, manifest/contracts/dependency checks, strict platform inventory, all 59 WebUI permutations, eight browser-conformance scenarios, and the complete focused workspace gate.

zemion closed this issue

2026-08-21 18:06:42 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-dataflow#11