GovOPlaN/govoplan-core
1
0
Fork 0
Code Issues 44 Pull Requests Actions Packages Projects Releases Wiki Activity

[Feature] Recurring datasource and data-transformation workflows #216

New Issue
Open
opened 2026-07-07 21:16:14 +02:00 by zemion · 4 comments
zemion commented 2026-07-07 21:16:14 +02:00
Owner
Copy Link

Model the monthly data extraction/transformation use case as a first-class product slice. The system should receive or fetch data, apply transformation rules, validate outputs, and prepare files for several downstream systems.

Source: docs/GOVERNMENT_OPERATIONS_VISION.md

Acceptance criteria:

  • decide whether govoplan-datasources and govoplan-dataflow repositories/modules are needed
  • define datasource, schema discovery, transformation rule, schedule, validation, output, and provenance contracts
  • support previewable diffs and diagnostics before publication
  • workflow can react to success, warning, failure, and manual approval states
  • audit records inputs, hashes, outputs, mapping versions, and operator actions
Model the monthly data extraction/transformation use case as a first-class product slice. The system should receive or fetch data, apply transformation rules, validate outputs, and prepare files for several downstream systems. Source: `docs/GOVERNMENT_OPERATIONS_VISION.md` Acceptance criteria: - decide whether `govoplan-datasources` and `govoplan-dataflow` repositories/modules are needed - define datasource, schema discovery, transformation rule, schedule, validation, output, and provenance contracts - support previewable diffs and diagnostics before publication - workflow can react to success, warning, failure, and manual approval states - audit records inputs, hashes, outputs, mapping versions, and operator actions
zemion commented 2026-07-28 00:58:12 +02:00
Author
Owner
Copy Link

The module decision and first work packages are now recorded under GovOPlaN/govoplan-dataflow#1, with source discovery in GovOPlaN/govoplan-connectors#9, comparison/review in GovOPlaN/govoplan-dataflow#7, and Workflow handoff in GovOPlaN/govoplan-workflow#4. This issue remains the Core-side cross-capability contract tracker.

The module decision and first work packages are now recorded under https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/1, with source discovery in https://git.add-ideas.de/GovOPlaN/govoplan-connectors/issues/9, comparison/review in https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/7, and Workflow handoff in https://git.add-ideas.de/GovOPlaN/govoplan-workflow/issues/4. This issue remains the Core-side cross-capability contract tracker. <!-- codex-recurring-dataflow-decomposition-2026-07-28 -->
zemion commented 2026-07-28 12:48:20 +02:00
Author
Owner
Copy Link

Codex State: progress

The architecture decision and base contracts are complete:

  • Connectors acquires external data and publishes origins.
  • Datasources governs live/cached/static sources, staging, immutable/frozen
    states, schema, fingerprints, and reads.
  • Dataflow consumes Datasources and records pinned fingerprints.
  • Workflow now has a separate reusable graph library and validation contract.
  • A composed check proves Connector -> Datasources -> Dataflow preview/lineage.

Implemented in 8b1910b, 1cb6228, 27302f0, 6ca3058, 85eef00, and meta
commit 163b35c.

Remaining here: recurring scheduling, quality/approval gates, output
publication, durable run lifecycle, and audit-complete Workflow handoff.
Datasource tracking starts at
GovOPlaN/govoplan-datasources#1.

## Codex State: progress The architecture decision and base contracts are complete: - Connectors acquires external data and publishes origins. - Datasources governs live/cached/static sources, staging, immutable/frozen states, schema, fingerprints, and reads. - Dataflow consumes Datasources and records pinned fingerprints. - Workflow now has a separate reusable graph library and validation contract. - A composed check proves Connector -> Datasources -> Dataflow preview/lineage. Implemented in `8b1910b`, `1cb6228`, `27302f0`, `6ca3058`, `85eef00`, and meta commit `163b35c`. Remaining here: recurring scheduling, quality/approval gates, output publication, durable run lifecycle, and audit-complete Workflow handoff. Datasource tracking starts at https://git.add-ideas.de/GovOPlaN/govoplan-datasources/issues/1.
zemion commented 2026-07-28 13:50:49 +02:00
Author
Owner
Copy Link

Codex State: progress

Summary

  • The repository/module split is now implemented: Connectors owns external origins, Datasources owns governed identities/materializations/publication, Dataflow owns transformation definitions/runs, and Workflow owns orchestration definitions.
  • Core exposes neutral Datasource publication and Dataflow run lifecycle contracts in baa2143; meta composition coverage is 5b79e7d.

Verification

  • Contract scan passed (47 modules, 48 providers, 24 requirements); version alignment passed for 67 repositories.

Next / Blocked

  • Keep open for previewable publication diffs, Workflow reaction to warning/review/failure states, asynchronous large-result execution, and full operator-action audit coverage.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - The repository/module split is now implemented: Connectors owns external origins, Datasources owns governed identities/materializations/publication, Dataflow owns transformation definitions/runs, and Workflow owns orchestration definitions. - Core exposes neutral Datasource publication and Dataflow run lifecycle contracts in baa2143; meta composition coverage is 5b79e7d. ### Verification - `Contract scan passed (47 modules, 48 providers, 24 requirements); version alignment passed for 67 repositories.` ### Next / Blocked - Keep open for previewable publication diffs, Workflow reaction to warning/review/failure states, asynchronous large-result execution, and full operator-action audit coverage. Suggested status label: `status/in-progress`
zemion commented 2026-07-28 15:06:42 +02:00
Author
Owner
Copy Link

The Datasources/Dataflow split is now implemented: governed Datasource publication plus reusable Dataflow definitions, manual/scheduled/event trigger records, immutable revisions, lineage, diagnostics, and output publication are available in govoplan-datasources@6294362 and govoplan-dataflow@08ddc8d. Workflow reaction/manual approval still depends on the Workflow instance runtime; production-scale asynchronous transformation remains govoplan-dataflow#10.

The Datasources/Dataflow split is now implemented: governed Datasource publication plus reusable Dataflow definitions, manual/scheduled/event trigger records, immutable revisions, lineage, diagnostics, and output publication are available in govoplan-datasources@6294362 and govoplan-dataflow@08ddc8d. Workflow reaction/manual approval still depends on the Workflow instance runtime; production-scale asynchronous transformation remains govoplan-dataflow#10.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/api
HTTP API contracts, routers, schemas, or API smoke behavior.
area/auth
Authentication, sessions, access bootstrap, or login behavior.
area/db
Database sessions, models, transactions, or persistence primitives.
area/devex
Local developer workflow, scripts, tests, tooling, or release helpers.
area/docs
Durable documentation and project guidance.
area/governance
Governance policy, audit, privacy, retention, or compliance behavior.
area/marketing
Public website, product messaging, publication copy, or legal page content.
area/migrations
Alembic migrations, schema bootstrap, or persistence evolution.
area/module-system
Module discovery, manifests, capabilities, routing, or optional integrations.
area/rbac
Permissions, roles, delegation, or authorization policy.
area/release
Versioning, release locks, tags, packaging, or dependency pins.
area/security
Security posture, static analysis, supply-chain hardening, or vulnerability remediation.
area/tenancy
Tenant boundaries, provisioning, or tenant-scoped data behavior.
area/webui
Shared WebUI shell, frontend components, routing, or frontend tests.
audit/complexity
Complexity finding from Radon, Xenon, or equivalent maintainability scans.
audit/duplication
Duplicated-code finding from jscpd or equivalent similarity scans.
audit/false-positive
Audit finding reviewed as a narrow false positive or acceptable risk.
audit/needs-design
Audit finding that needs an architectural or product decision before implementation.
audit/quick-fix
Audit finding that appears narrow and directly fixable.
audit/structural
Audit finding that needs design, refactoring, or behavior review.
codex/needs-human
Needs an explicit human decision before Codex should implement.
codex/ready
Suitable for Codex to pick up with the existing issue context.
module/access
GovOPlaN access, identity, authentication, RBAC, and administration behavior.
module/addresses
GovOPlaN Addresses module behavior or integration.
module/admin
GovOPlaN Admin module behavior or integration.
module/appointments
GovOPlaN Appointments module behavior or integration.
module/audit
GovOPlaN Audit module behavior or integration.
module/calendar
GovOPlaN Calendar module behavior or integration.
module/campaign
GovOPlaN campaign module behavior or integration.
module/cases
GovOPlaN Cases module behavior or integration.
module/connectors
GovOPlaN Connectors module behavior or integration.
module/core
GovOPlaN core runner, shared primitives, shell, or extension points.
module/dataflow
GovOPlaN Dataflow module behavior or integration.
module/datasources
GovOPlaN governed datasource contracts, catalogs, and integrations.
module/dist-lists
GovOPlaN Distribution Lists module behavior or integration.
module/dms
GovOPlaN Dms module behavior or integration.
module/erp
GovOPlaN Erp module behavior or integration.
module/evaluation
GovOPlaN Evaluation module behavior or integration.
module/files
GovOPlaN files module behavior or integration.
module/fit-connect
GovOPlaN Fit Connect module behavior or integration.
module/forms
GovOPlaN Forms module behavior or integration.
module/identity
GovOPlaN Identity module behavior or integration.
module/identity-trust
GovOPlaN Identity Trust module behavior or integration.
module/idm
GovOPlaN Idm module behavior or integration.
module/ledger
GovOPlaN Ledger module behavior or integration.
module/mail
GovOPlaN mail module behavior or integration.
module/notifications
GovOPlaN Notifications module behavior or integration.
module/ops
GovOPlaN Ops module behavior or integration.
module/organizations
GovOPlaN Organizations module behavior or integration.
module/payments
GovOPlaN Payments module behavior or integration.
module/permits
GovOPlaN Permits module behavior or integration.
module/policy
GovOPlaN Policy module behavior or integration.
module/poll
GovOPlaN Poll module behavior or integration.
module/portal
GovOPlaN Portal module behavior or integration.
module/postbox
GovOPlaN Postbox module behavior or integration.
module/reporting
GovOPlaN Reporting module behavior or integration.
module/risk-compliance
GovOPlaN Risk Compliance module behavior or integration.
module/scheduling
GovOPlaN Scheduling module behavior or integration.
module/search
GovOPlaN Search module behavior or integration.
module/tasks
GovOPlaN Tasks module behavior or integration.
module/templates
GovOPlaN Templates module behavior or integration.
module/tenancy
GovOPlaN Tenancy module behavior or integration.
module/views
GovOPlaN governed task views, interface projections, and workflow view integration.
module/workflow
GovOPlaN Workflow module behavior or integration.
module/xoev
GovOPlaN Xoev module behavior or integration.
module/xrechnung
GovOPlaN Xrechnung module behavior or integration.
module/xta-osci
GovOPlaN Xta Osci module behavior or integration.
priority
p0
Immediate stop-the-line priority.
priority
p1
High priority for the next focused work window.
priority
p2
Normal planned priority.
priority
p3
Low priority or opportunistic cleanup.
source/backlog-import
Imported from markdown backlog, roadmap, plan, or TODO files.
source/security-audit
Created from a structured security or code-quality audit report.
source/todo-scan
Imported from inline TODO/FIXME/HACK markers by the Gitea TODO importer.
status
blocked
Cannot progress without a decision, dependency, credential, or external change.
status
in-progress
Currently being worked.
status
needs-info
Needs clarifying input before implementation can proceed safely.
status
ready
Ready for implementation.
status
triage
Needs review, ownership, priority, or acceptance criteria.
type
bug
A reproducible defect, regression, or incorrect behavior.
type
debt
Cleanup, refactoring, risk reduction, or deferred engineering work.
type
docs
Documentation, process, or developer workflow work.
type
feature
New user-visible behavior or platform capability.
type
task
Implementation, maintenance, migration, or operational work.
type
user-story
End-to-end user journey or real-world process story used to steer product slices.
No Label
area/api
area/governance
codex/ready
module/dataflow
module/datasources
module/workflow
priority
p1
status
in-progress
type
feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
zemion
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-core#216
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?