[User story] guided monthly data workflow from incoming files to campaign send #8

New Issue

2026-07-14T14:01:26+02:00

zemion commented

2026-07-14 14:01:26 +02:00

User Story

As an operator responsible for a recurring monthly data process, I want GovOPlaN to guide me through receiving structured files, importing and comparing data, correcting and rerunning processing steps, and finally producing and sending campaign output, so that the process becomes traceable, resumable, configurable, and less error-prone.

Real Process Context

Monthly files arrive with structured information. The current process has three major branches:

Part of the structured information must be imported into an existing database.
Another part must be processed and compared against data from a second database. The comparison needs an interactive UI for review/decisions and an Excel export of the comparison result.
Another part must be processed and linked with existing database records. This is currently a run -> verify -> correct -> rerun loop. The final result is a large bulk load of files that is sent through the campaign module.

The preferred path is direct database access from GovOPlaN. A fallback path should support static exports/imports when direct access is not possible.

Product Shape

The target experience is an interactive workflow that can be configured and then executed by users. It should guide operators through the case-specific process, preserve progress, allow leaving and resuming, surface open workflow tasks in a reminder/widget, and hand off to other modules when needed.

For example, a workflow step may create or prepare the necessary campaign configuration, route the user to the campaign module, frame the campaign work as part of the active workflow, wait for successful sending, and then continue the workflow.

Module Touchpoints

Likely modules involved:

module/workflow: owns workflow definition, execution state, resumability, task framing, and cross-module orchestration.
module/files: receives monthly files, stores source artifacts, generated artifacts, exports, and bulk output files.
module/connectors: direct database connections and external source connectors where feasible.
module/campaign: campaign setup, generated recipients/assets, send handoff, send status callback.
module/reporting: comparison views and exportable result output, especially Excel-style exports.
module/notifications: reminders for open workflow tasks and completed async steps.
module/tasks: optional personal/team task projection for workflow steps that require human action.

Potential missing or future module concepts:

datasource: database/static-export source definitions, credentials, schema mapping, import contracts.
dataflow: reusable transformation/comparison/linking pipelines and step outputs.

These may be separate modules later, or initially capabilities owned by workflow/connectors/reporting until boundaries become clearer.

Acceptance Criteria

A workflow can be configured as a sequence of typed steps, including file input, database/static-source input, processing, comparison, manual review, export, campaign preparation, campaign send handoff, verification, and completion.
A workflow execution has durable state and can be left and resumed later.
A user can see open workflow tasks and return to the active execution context from a widget or reminder surface.
A workflow step can call module capabilities without hard module-to-module imports, using manifest/capability contracts.
A comparison step can present interactive review decisions and export results to Excel-compatible output.
A processing/linking step can support a run -> verify -> correct -> rerun loop with source/version/provenance tracking.
A campaign handoff step can create or prepare campaign configuration, guide the user into campaign, and wait for a successful send/result before continuing.
Direct database querying is supported when configured; static exports remain a fallback path.
Every imported/generated artifact is traceable back to input files, source queries/exports, workflow execution, and user decisions.

Open Decisions

Whether datasource/dataflow should become first-class modules immediately or start as workflow/connectors/reporting capabilities.
How much of the workflow step UI is generic versus owned by each participating module.
How power-user configuration should be represented: declarative workflow definitions, low-code UI, code-backed plugins, or a hybrid.
Which database types and static export formats should be supported first.
How to model provenance, reruns, corrections, and auditability across files, workflow, and campaign.

First Vertical Slice Candidate

A pragmatic first slice could be:

Upload or select a monthly source file from files.
Configure a small structured import/mapping step.
Run a comparison against a static second source.
Show an interactive comparison table with accept/reject/correct decisions.
Export the comparison result to Excel.
Persist workflow state and show the open workflow in a reminder/task surface.

Campaign handoff can be the second slice once the state model and step UI are proven.

## User Story As an operator responsible for a recurring monthly data process, I want GovOPlaN to guide me through receiving structured files, importing and comparing data, correcting and rerunning processing steps, and finally producing and sending campaign output, so that the process becomes traceable, resumable, configurable, and less error-prone. ## Real Process Context Monthly files arrive with structured information. The current process has three major branches: 1. Part of the structured information must be imported into an existing database. 2. Another part must be processed and compared against data from a second database. The comparison needs an interactive UI for review/decisions and an Excel export of the comparison result. 3. Another part must be processed and linked with existing database records. This is currently a run -> verify -> correct -> rerun loop. The final result is a large bulk load of files that is sent through the campaign module. The preferred path is direct database access from GovOPlaN. A fallback path should support static exports/imports when direct access is not possible. ## Product Shape The target experience is an interactive workflow that can be configured and then executed by users. It should guide operators through the case-specific process, preserve progress, allow leaving and resuming, surface open workflow tasks in a reminder/widget, and hand off to other modules when needed. For example, a workflow step may create or prepare the necessary campaign configuration, route the user to the campaign module, frame the campaign work as part of the active workflow, wait for successful sending, and then continue the workflow. ## Module Touchpoints Likely modules involved: - `module/workflow`: owns workflow definition, execution state, resumability, task framing, and cross-module orchestration. - `module/files`: receives monthly files, stores source artifacts, generated artifacts, exports, and bulk output files. - `module/connectors`: direct database connections and external source connectors where feasible. - `module/campaign`: campaign setup, generated recipients/assets, send handoff, send status callback. - `module/reporting`: comparison views and exportable result output, especially Excel-style exports. - `module/notifications`: reminders for open workflow tasks and completed async steps. - `module/tasks`: optional personal/team task projection for workflow steps that require human action. Potential missing or future module concepts: - datasource: database/static-export source definitions, credentials, schema mapping, import contracts. - dataflow: reusable transformation/comparison/linking pipelines and step outputs. These may be separate modules later, or initially capabilities owned by workflow/connectors/reporting until boundaries become clearer. ## Acceptance Criteria - A workflow can be configured as a sequence of typed steps, including file input, database/static-source input, processing, comparison, manual review, export, campaign preparation, campaign send handoff, verification, and completion. - A workflow execution has durable state and can be left and resumed later. - A user can see open workflow tasks and return to the active execution context from a widget or reminder surface. - A workflow step can call module capabilities without hard module-to-module imports, using manifest/capability contracts. - A comparison step can present interactive review decisions and export results to Excel-compatible output. - A processing/linking step can support a run -> verify -> correct -> rerun loop with source/version/provenance tracking. - A campaign handoff step can create or prepare campaign configuration, guide the user into campaign, and wait for a successful send/result before continuing. - Direct database querying is supported when configured; static exports remain a fallback path. - Every imported/generated artifact is traceable back to input files, source queries/exports, workflow execution, and user decisions. ## Open Decisions - Whether datasource/dataflow should become first-class modules immediately or start as workflow/connectors/reporting capabilities. - How much of the workflow step UI is generic versus owned by each participating module. - How power-user configuration should be represented: declarative workflow definitions, low-code UI, code-backed plugins, or a hybrid. - Which database types and static export formats should be supported first. - How to model provenance, reruns, corrections, and auditability across files, workflow, and campaign. ## First Vertical Slice Candidate A pragmatic first slice could be: 1. Upload or select a monthly source file from files. 2. Configure a small structured import/mapping step. 3. Run a comparison against a static second source. 4. Show an interactive comparison table with accept/reject/correct decisions. 5. Export the comparison result to Excel. 6. Persist workflow state and show the open workflow in a reminder/task surface. Campaign handoff can be the second slice once the state model and step UI are proven.

zemion referenced this issue from GovOPlaN/govoplan-dataflow

2026-07-28 00:58:08 +02:00

[Epic] Deliver a governed Dataflow manager with graphical and SQL editing #1

zemion referenced this issue from GovOPlaN/govoplan-connectors

2026-07-28 00:58:09 +02:00

[Feature] Provide governed tabular datasource discovery and bounded preview #9

zemion referenced this issue from GovOPlaN/govoplan-workflow

2026-07-28 00:58:11 +02:00

[Feature] Orchestrate resumable Dataflow runs and human review handoffs #4

zemion commented

2026-07-28 00:58:11 +02:00

Product decision on 2026-07-28: activate the dedicated Dataflow boundary and begin the first vertical slice.

Prioritized work packages:

P1 Dataflow implementation epic: GovOPlaN/govoplan-dataflow#1
P1 connector-backed tabular source contract: GovOPlaN/govoplan-connectors#9
P1 comparison/reconciliation transforms: GovOPlaN/govoplan-dataflow#7
P2 resumable Workflow handoff: GovOPlaN/govoplan-workflow#4
P2 governed Reporting output: GovOPlaN/govoplan-reporting#2

A separate govoplan-datasources repository is not created yet. The first source catalog and schema-discovery contract belongs to Connectors; this decision should be revisited only when provider-neutral source governance demonstrably outgrows connector ownership.

Product decision on 2026-07-28: activate the dedicated Dataflow boundary and begin the first vertical slice. Prioritized work packages: - P1 Dataflow implementation epic: https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/1 - P1 connector-backed tabular source contract: https://git.add-ideas.de/GovOPlaN/govoplan-connectors/issues/9 - P1 comparison/reconciliation transforms: https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/7 - P2 resumable Workflow handoff: https://git.add-ideas.de/GovOPlaN/govoplan-workflow/issues/4 - P2 governed Reporting output: https://git.add-ideas.de/GovOPlaN/govoplan-reporting/issues/2 A separate `govoplan-datasources` repository is not created yet. The first source catalog and schema-discovery contract belongs to Connectors; this decision should be revisited only when provider-neutral source governance demonstrably outgrows connector ownership.

zemion added the area/governance codex/ready module/campaign module/dataflow module/datasources module/files module/workflow

labels 2026-07-29 14:15:06 +02:00

zemion commented

2026-07-29 15:52:56 +02:00

Codex State: progress

Summary

Added an executable monthly-reconciliation golden flow with separate incoming/existing datasets, conversion, quality checks, reconciliation, and exact expected output.

Changed Files

govoplan-dataflow/fixtures/golden/monthly-reconciliation

Verification

Golden-flow fixture test passed

Next / Blocked

Extend the fixture through durable human decisions, database import/publication, Excel export, workflow framing, and campaign handoff.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Added an executable monthly-reconciliation golden flow with separate incoming/existing datasets, conversion, quality checks, reconciliation, and exact expected output. ### Changed Files - `govoplan-dataflow/fixtures/golden/monthly-reconciliation` ### Verification - `Golden-flow fixture test passed` ### Next / Blocked - Extend the fixture through durable human decisions, database import/publication, Excel export, workflow framing, and campaign handoff. Suggested status label: `status/in-progress`

zemion added

and removed

labels 2026-07-29 15:54:02 +02:00

zemion commented

2026-07-30 21:20:02 +02:00

Decision cleanup, 2026-07-30

The architecture questions in the original body are no longer blockers:

Datasources and Dataflow are first-class modules; Connectors owns protocol adapters, Datasources owns governed source/snapshot semantics, and Dataflow owns transformations and execution.
Workflow owns generic orchestration, resumability and action/effect state; each participating module owns its handoff UI and domain effects behind versioned capabilities.
Power-user configuration is hybrid: graphical definitions plus SQL/typed expressions and versioned provider/plugin contracts where needed.
Provenance pins source/revision/hash, inputs, decisions and outputs; reruns are new attributable executions rather than mutation of historical evidence.

Remaining work is implementation tracked by the linked Dataflow, Workflow, Reporting, Campaign and connector tickets. Concrete provider priorities and format coverage can be chosen incrementally per connector rather than blocking this user story.

## Decision cleanup, 2026-07-30 The architecture questions in the original body are no longer blockers: - Datasources and Dataflow are first-class modules; Connectors owns protocol adapters, Datasources owns governed source/snapshot semantics, and Dataflow owns transformations and execution. - Workflow owns generic orchestration, resumability and action/effect state; each participating module owns its handoff UI and domain effects behind versioned capabilities. - Power-user configuration is hybrid: graphical definitions plus SQL/typed expressions and versioned provider/plugin contracts where needed. - Provenance pins source/revision/hash, inputs, decisions and outputs; reruns are new attributable executions rather than mutation of historical evidence. Remaining work is implementation tracked by the linked Dataflow, Workflow, Reporting, Campaign and connector tickets. Concrete provider priorities and format coverage can be chosen incrementally per connector rather than blocking this user story.

zemion changed title from ~~User story: guided monthly data workflow from incoming files to campaign send~~ to [User story] guided monthly data workflow from incoming files to campaign send

2026-07-31 01:43:35 +02:00

zemion commented

2026-08-04 12:20:36 +02:00

Codex State: progress

Summary

The monthly-data reference flow now has immutable Connector snapshots, governed Datasource staging/publication, pinned Dataflow outputs, deterministic quality gates, and version-pinned human reconciliation decisions.
The end-to-end composition test verifies Connector acquisition through a queued Dataflow run into a frozen output materialization.

Verification

Full focused platform check passed, including 50 WebUI module permutations and full-product bundle budgets.

Next / Blocked

Remaining vertical slices: decision authoring UI, Excel-ready output, Workflow-guided handoff/resume around Campaign, and production database/connector fixtures.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - The monthly-data reference flow now has immutable Connector snapshots, governed Datasource staging/publication, pinned Dataflow outputs, deterministic quality gates, and version-pinned human reconciliation decisions. - The end-to-end composition test verifies Connector acquisition through a queued Dataflow run into a frozen output materialization. ### Verification - `Full focused platform check passed, including 50 WebUI module permutations and full-product bundle budgets.` ### Next / Blocked - Remaining vertical slices: decision authoring UI, Excel-ready output, Workflow-guided handoff/resume around Campaign, and production database/connector fixtures. Suggested status label: `status/in-progress`

zemion commented

2026-08-04 12:50:31 +02:00

Codex State: progress

Summary

The monthly-data vertical can now carry an exact successful Dataflow run into Reporting through a pinned immutable Datasource materialization.
Reporting receives source lineage and governance evidence without rerunning the flow, and both Dataflow and Datasource access are rechecked at read time.

Verification

Full focused platform check passed, including the Connector -> Datasource -> pinned Dataflow composition and 50 WebUI module permutations.

Next / Blocked

Remaining product slices are reconciliation decision authoring UI, Excel-ready interaction/export proof, Workflow-guided Campaign handoff/resume, and production database/file connector fixtures.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - The monthly-data vertical can now carry an exact successful Dataflow run into Reporting through a pinned immutable Datasource materialization. - Reporting receives source lineage and governance evidence without rerunning the flow, and both Dataflow and Datasource access are rechecked at read time. ### Verification - `Full focused platform check passed, including the Connector -> Datasource -> pinned Dataflow composition and 50 WebUI module permutations.` ### Next / Blocked - Remaining product slices are reconciliation decision authoring UI, Excel-ready interaction/export proof, Workflow-guided Campaign handoff/resume, and production database/file connector fixtures. Suggested status label: `status/in-progress`

zemion commented

2026-08-06 12:47:34 +02:00

Codex State: progress

The monthly-data track now has durable human reconciliation decisions, stale-input invalidation, bounded decision-source reuse and a tested exact-run formula-safe CSV publication path for Excel. The existing monthly golden fixture and Connector -> Datasource -> Dataflow composition remain green.

Commits: govoplan-dataflow@b14c693, govoplan-reporting@c2c400a, govoplan@69ba103.

The story remains open for the complete resumable Workflow framing, incoming-file operational path and Campaign handoff/send evidence.

## Codex State: progress The monthly-data track now has durable human reconciliation decisions, stale-input invalidation, bounded decision-source reuse and a tested exact-run formula-safe CSV publication path for Excel. The existing monthly golden fixture and Connector -> Datasource -> Dataflow composition remain green. Commits: `govoplan-dataflow@b14c693`, `govoplan-reporting@c2c400a`, `govoplan@69ba103`. The story remains open for the complete resumable Workflow framing, incoming-file operational path and Campaign handoff/send evidence.

zemion commented

2026-08-06 14:12:53 +02:00

The normalized source ideas and user-story orientation are now preserved in govoplan/docs/PRODUCT_INPUT_REGISTER.md. This issue remains the canonical live work item for the corresponding outcome.

The normalized source ideas and user-story orientation are now preserved in `govoplan/docs/PRODUCT_INPUT_REGISTER.md`. This issue remains the canonical live work item for the corresponding outcome.

zemion commented

2026-08-06 16:17:46 +02:00

Codex State: progress

Revalidated the monthly-data composition after adding native Datasources search coverage and the shared durable-work contract. Immutable acquisition, governed transformation/reconciliation, reporting/export handoff, and composition checks pass under the consolidated focused gate. The story remains open for the remaining target-provider and product-level browser/operational evidence.

## Codex State: progress Revalidated the monthly-data composition after adding native Datasources search coverage and the shared durable-work contract. Immutable acquisition, governed transformation/reconciliation, reporting/export handoff, and composition checks pass under the consolidated focused gate. The story remains open for the remaining target-provider and product-level browser/operational evidence.

zemion referenced this issue

2026-08-21 02:02:09 +02:00

[Privacy] Complete DSAR provider coverage across data-owning modules #47

zemion referenced this issue

2026-08-21 03:49:35 +02:00

[Privacy] Complete DSAR provider coverage across data-owning modules #47

zemion referenced this issue

2026-09-08 05:27:54 +02:00

[Roadmap] Deliver governed dataflows, search, and established-tool integration #19

Sign in to join this conversation.