[Feature] Publish module outputs as immutable datasource materializations #3

New Issue

2026-07-28T12:48:19+02:00

zemion commented

2026-07-28 12:48:19 +02:00

Parent: #1

Define a producer-side lifecycle contract so Dataflow and other modules can
publish outputs without importing Datasources internals.

Acceptance criteria:

versioned stage/publish DTO and capability for module-owned producers
idempotency key and producer/run/revision provenance
bounded inline output plus durable artifact/object reference support
atomic publication of materialization metadata and current-state change
configurable frozen/publication label
audit event and notification-ready terminal state
Dataflow publishes a pinned run output and returns a datasource/materialization reference
Workflow can wait for publication success, warning, review-required, or failure

Parent: https://git.add-ideas.de/GovOPlaN/govoplan-datasources/issues/1 Define a producer-side lifecycle contract so Dataflow and other modules can publish outputs without importing Datasources internals. Acceptance criteria: - versioned stage/publish DTO and capability for module-owned producers - idempotency key and producer/run/revision provenance - bounded inline output plus durable artifact/object reference support - atomic publication of materialization metadata and current-state change - configurable frozen/publication label - audit event and notification-ready terminal state - Dataflow publishes a pinned run output and returns a datasource/materialization reference - Workflow can wait for publication success, warning, review-required, or failure

zemion added the module/dataflow module/datasources area/api module/workflow area/db codex/ready

labels 2026-07-28 12:48:19 +02:00

zemion referenced this issue from GovOPlaN/govoplan-dataflow

2026-07-28 12:48:21 +02:00

[Feature] Expose Dataflow run lifecycle to Workflow #9

zemion commented

2026-07-28 13:49:46 +02:00

Codex State: progress

Summary

Implemented the versioned datasources.publication capability and SQL provider in 8ae7dac; Core contract is baa2143.
Producer module/run provenance, idempotent replay and conflict detection, immutable materializations, optional frozen labels, existing/new targets, and returned datasource/materialization references are covered.

Verification

Datasource tests: 10 passed.
Connector -> Datasources -> pinned Dataflow publication composition passed in govoplan 5b79e7d.

Next / Blocked

Keep open for durable artifact/object references, high-concurrency idempotency/revision hardening, generic audit/notification emission, and Workflow terminal-state consumption.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Implemented the versioned datasources.publication capability and SQL provider in 8ae7dac; Core contract is baa2143. - Producer module/run provenance, idempotent replay and conflict detection, immutable materializations, optional frozen labels, existing/new targets, and returned datasource/materialization references are covered. ### Verification - `Datasource tests: 10 passed.` - `Connector -> Datasources -> pinned Dataflow publication composition passed in govoplan 5b79e7d.` ### Next / Blocked - Keep open for durable artifact/object references, high-concurrency idempotency/revision hardening, generic audit/notification emission, and Workflow terminal-state consumption. Suggested status label: `status/in-progress`

zemion added

and removed

labels 2026-07-28 13:51:38 +02:00

zemion commented

2026-08-04 12:19:46 +02:00

Codex State: progress

Summary

Producer publication now applies deterministic governance quality/schema gates before any catalogue side effect and retains validation evidence on the immutable materialization.
Successful publication emits transactional terminal event evidence; the composition fixture now verifies Connector -> Datasource -> pinned Dataflow publication.
Origin registration and refresh preserve source mode, health, pushdown, and bounded-preview evidence in commit 067c827.

Verification

27 Datasources tests passed (1 PostgreSQL-only skip); full focused platform check and Datasources WebUI typecheck passed.

Next / Blocked

Remaining: durable artifact-reference publication for outputs too large for inline rows, explicit warning/review wait states for Workflow, and broader high-concurrency publication evidence.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Producer publication now applies deterministic governance quality/schema gates before any catalogue side effect and retains validation evidence on the immutable materialization. - Successful publication emits transactional terminal event evidence; the composition fixture now verifies Connector -> Datasource -> pinned Dataflow publication. - Origin registration and refresh preserve source mode, health, pushdown, and bounded-preview evidence in commit 067c827. ### Verification - `27 Datasources tests passed (1 PostgreSQL-only skip); full focused platform check and Datasources WebUI typecheck passed.` ### Next / Blocked - Remaining: durable artifact-reference publication for outputs too large for inline rows, explicit warning/review wait states for Workflow, and broader high-concurrency publication evidence. Suggested status label: `status/in-progress`

zemion commented

2026-08-04 12:50:23 +02:00

Codex State: progress

Summary

Concurrent publications with the same tenant, producer, and idempotency identity are serialized by a PostgreSQL transaction advisory lock and converge on one publication/materialization.
Replay remains deterministic for committed results; a two-session PostgreSQL regression test covers the race.

Changed Files

src/govoplan_datasources/backend/service.py
tests/test_postgres_materialization_concurrency.py

Verification

Datasources: 28 passed, 2 PostgreSQL-environment skips.
Full focused platform check passed, including 50 WebUI module permutations and the full-product build.

Next / Blocked

Remaining acceptance work: durable object/artifact-backed output publication and explicit Workflow warning/review-required terminal-state consumption.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Concurrent publications with the same tenant, producer, and idempotency identity are serialized by a PostgreSQL transaction advisory lock and converge on one publication/materialization. - Replay remains deterministic for committed results; a two-session PostgreSQL regression test covers the race. ### Changed Files - `src/govoplan_datasources/backend/service.py` - `tests/test_postgres_materialization_concurrency.py` ### Verification - `Datasources: 28 passed, 2 PostgreSQL-environment skips.` - `Full focused platform check passed, including 50 WebUI module permutations and the full-product build.` ### Next / Blocked - Remaining acceptance work: durable object/artifact-backed output publication and explicit Workflow warning/review-required terminal-state consumption. Suggested status label: `status/in-progress`

zemion commented

2026-08-21 18:03:41 +02:00

Implemented and verified across Core fbea74a, Datasources 97ca670, and Dataflow a86220d; released in Datasources as v0.1.19.

Acceptance evidence:

The publication contract now supports exactly one of inline rows or a durable, checksum-bound artifact reference.
Artifact backends are optional Core capabilities with tenant-aware verify/read/delete operations; Datasources persists provider-neutral references and performs bounded reads.
Immutable materializations now have typed terminal outcomes: published, published_with_warnings, and review_required.
Content-rule evidence is bound to artifact checksum and policy hash. Missing/deferred evidence creates a reviewable immutable result without promoting it as current; invalid evidence creates no catalogue side effect.
Workflow/Dataflow receive warning/review outcomes and hand them to the existing review path.
User/admin documentation and exact contextual help were updated.

Verification: 42 Datasources tests (2 environment-dependent PostgreSQL skips), 92 Dataflow tests, Workflow lifecycle tests, Ruff, TypeScript, manifest/contract/dependency checks, strict inventory, all 59 WebUI permutations, eight browser conformance scenarios, and the complete workspace focused gate passed.

Implemented and verified across Core `fbea74a`, Datasources `97ca670`, and Dataflow `a86220d`; released in Datasources as `v0.1.19`. Acceptance evidence: - The publication contract now supports exactly one of inline rows or a durable, checksum-bound artifact reference. - Artifact backends are optional Core capabilities with tenant-aware verify/read/delete operations; Datasources persists provider-neutral references and performs bounded reads. - Immutable materializations now have typed terminal outcomes: `published`, `published_with_warnings`, and `review_required`. - Content-rule evidence is bound to artifact checksum and policy hash. Missing/deferred evidence creates a reviewable immutable result without promoting it as current; invalid evidence creates no catalogue side effect. - Workflow/Dataflow receive warning/review outcomes and hand them to the existing review path. - User/admin documentation and exact contextual help were updated. Verification: 42 Datasources tests (2 environment-dependent PostgreSQL skips), 92 Dataflow tests, Workflow lifecycle tests, Ruff, TypeScript, manifest/contract/dependency checks, strict inventory, all 59 WebUI permutations, eight browser conformance scenarios, and the complete workspace focused gate passed.

zemion closed this issue

2026-08-21 18:03:41 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-datasources#3