GovOPlaN/govoplan-core
2
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

[Integrity review] Implement 2026-09-08 data-integrity and performance findings #298

New Issue
Closed
opened 2026-09-08 10:24:50 +02:00 by zemion · 2 comments
zemion commented 2026-09-08 10:24:50 +02:00
Owner
Copy Link
Copy Source

Cross-module tracking for the authorized 79-repository review implementation.

Core owns I6 shared CSV text/source-evidence contracts, P1 narrow DataGrid index lookup, P6 shell request coalescing/generation fences, P8 bounded conditional JSON buffering, and regression-backed keyed-list merge order handling. Feature behavior stays module-owned. Selectively consolidate the eight non-migration clone groups after equivalence checks; historical migrations remain unchanged.

Acceptance: failure-specific tests, tenant/permission equivalence, source/output integrity, query/work-count measurements, owning EN/DE documentation, manifest-shape and cross-module focused checks. Production load tests remain a separate operational check.

User authorized implementation on 2026-09-08. Work is local until explicitly published; keep this issue open until verification and publication state are recorded. Preserve tenant isolation, current authorization, immutable history, audit evidence, deterministic ordering, retry safety and existing API compatibility. Add focused regressions and owning EN/DE documentation. No live mail/provider side effects or historical data rewrites.

Owning issues

Cross-module tracking for the authorized 79-repository review implementation. Core owns I6 shared CSV text/source-evidence contracts, P1 narrow DataGrid index lookup, P6 shell request coalescing/generation fences, P8 bounded conditional JSON buffering, and regression-backed keyed-list merge order handling. Feature behavior stays module-owned. Selectively consolidate the eight non-migration clone groups after equivalence checks; historical migrations remain unchanged. Acceptance: failure-specific tests, tenant/permission equivalence, source/output integrity, query/work-count measurements, owning EN/DE documentation, manifest-shape and cross-module focused checks. Production load tests remain a separate operational check. User authorized implementation on 2026-09-08. Work is local until explicitly published; keep this issue open until verification and publication state are recorded. Preserve tenant isolation, current authorization, immutable history, audit evidence, deterministic ordering, retry safety and existing API compatibility. Add focused regressions and owning EN/DE documentation. No live mail/provider side effects or historical data rewrites. ## Owning issues - https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/23 - https://git.add-ideas.de/GovOPlaN/govoplan-addresses/issues/25 - https://git.add-ideas.de/GovOPlaN/govoplan-calendar/issues/25 - https://git.add-ideas.de/GovOPlaN/govoplan-mail/issues/24 - https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/47 - https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/101 - https://git.add-ideas.de/GovOPlaN/govoplan-tickets/issues/2 - https://git.add-ideas.de/GovOPlaN/govoplan-cases/issues/7 - https://git.add-ideas.de/GovOPlaN/govoplan-connectors/issues/19 - https://git.add-ideas.de/GovOPlaN/govoplan-datasources/issues/9
zemion added the
type
bug
priority
p1
status
in-progress
 labels 2026-09-08 10:24:50 +02:00
zemion commented 2026-09-08 11:51:54 +02:00
Author
Owner
Copy Link
Copy Source

Local implementation receipt — 2026-09-08

All 14 named review findings plus the latent keyed-list insertion-order defect and eight reviewed non-migration clone groups are implemented locally across 16 repositories. Core adds bounded conditional JSON buffering, exact SQL JSON predicates, CSV source/projection contracts, stable authority keys, single-flight module refresh and DataGrid first-occurrence indices. Shared helpers preserve owner-specific authority/normalization semantics. Independent follow-up review also fixed additional CSV policy/evidence, frontend authority, Files provenance, Address before-image and Calendar amplification edge cases. Core targeted 36 tests, independently rerun middleware/merge 14 tests and ordering/byte-boundary probes, DataGrid 6400 equivalence comparisons and 7 Playwright tests pass; frontend save/detail/auth regressions 26 pass. English/German platform and owning-module documentation updated.

Verification: tools/checks/check-focused.sh completed successfully,72/72 manifest architecture checks passed, and changed-source Ruff F plus git diff --check passed across 16 repositories. Tests use isolated databases/mock providers; no live data/provider mutation, commit, push, tag, deployment or release was performed. Changes remain local and unpublished: keep this issue OPEN until publication/rollout state is recorded. Before rollout, back up and test coordinated Mail/Files/Connectors/Datasources migrations on a database copy. PostgreSQL concurrency, real provider races and representative load testing remain operational verification, not a claimed certification.

Load-related verification note: an extra parallel Dataflow run hit a DuckDB wall-clock timeout during frontend production builds. After all build/browser work finished, the serialized final Dataflow suite passed all 117 tests without changing worker safety limits. Deployment capacity/deadline testing remains necessary.

## Local implementation receipt — 2026-09-08 All 14 named review findings plus the latent keyed-list insertion-order defect and eight reviewed non-migration clone groups are implemented locally across 16 repositories. Core adds bounded conditional JSON buffering, exact SQL JSON predicates, CSV source/projection contracts, stable authority keys, single-flight module refresh and DataGrid first-occurrence indices. Shared helpers preserve owner-specific authority/normalization semantics. Independent follow-up review also fixed additional CSV policy/evidence, frontend authority, Files provenance, Address before-image and Calendar amplification edge cases. Core targeted 36 tests, independently rerun middleware/merge 14 tests and ordering/byte-boundary probes, DataGrid 6400 equivalence comparisons and 7 Playwright tests pass; frontend save/detail/auth regressions 26 pass. English/German platform and owning-module documentation updated. Verification: tools/checks/check-focused.sh completed successfully,72/72 manifest architecture checks passed, and changed-source Ruff F plus git diff --check passed across 16 repositories. Tests use isolated databases/mock providers; no live data/provider mutation, commit, push, tag, deployment or release was performed. Changes remain local and unpublished: keep this issue OPEN until publication/rollout state is recorded. Before rollout, back up and test coordinated Mail/Files/Connectors/Datasources migrations on a database copy. PostgreSQL concurrency, real provider races and representative load testing remain operational verification, not a claimed certification. Load-related verification note: an extra parallel Dataflow run hit a DuckDB wall-clock timeout during frontend production builds. After all build/browser work finished, the serialized final Dataflow suite passed all 117 tests without changing worker safety limits. Deployment capacity/deadline testing remains necessary.
zemion commented 2026-09-08 12:57:27 +02:00
Author
Owner
Copy Link
Copy Source

Source publication and database verification — 2026-09-08

Remote main branches and annotated tags were verified against the exact coordinated commits. The implementation-specific tests and EN/DE documentation recorded above remain the acceptance evidence. Cross-module focused checks, manifest shapes, 63 WebUI build configurations and 230 browser tests passed. This records source publication, not a package-registry release, runtime image, website deployment or production rollout.

Database correction to the previous local-only note: the already-running development server had automatically applied Files a2b3c4d5e701, Mail b5d6e7f8091a, Connectors d2a4c6e8f0b1 and Datasources e2b8d4a0f6c3 during reloads. Their heads and expected schema/indexes were verified in the local development PostgreSQL database. A current-state backup was captured and restored into a private, isolated PostgreSQL 16 cluster; this was not a pre-upgrade backup. The supported no-op migration preserved all 281 tables and 142,287 rows by per-row SHA-256 comparison, and retained the same migration heads. Files source identity recomputation matched all 7,385 rows (2 non-null identities, zero mismatches). Both Datasources PostgreSQL concurrency tests, previously skipped without a dedicated database, passed against that isolated cluster. The test cluster was stopped. No real mail delivery, POP3 retrieval/deletion, legacy mailbox reconciliation or external connector import was initiated.

Representative production load/deadline testing and real-provider race/compatibility verification remain separate operational rollout checks; this is not security certification. Operational follow-up: #299.

Closing this implementation issue: its code, regression, documentation and source-publication criteria are satisfied. Operational checks above are not claimed complete.

## Source publication and database verification — 2026-09-08 - govoplan: [v0.1.46](https://git.add-ideas.de/GovOPlaN/govoplan/src/tag/v0.1.46) · [14b19fbead79](https://git.add-ideas.de/GovOPlaN/govoplan/commit/14b19fbead79ebf65ef9b49ed82790a9ab214b3c) - govoplan-addresses: [v0.1.23](https://git.add-ideas.de/GovOPlaN/govoplan-addresses/src/tag/v0.1.23) · [d4fa024034e5](https://git.add-ideas.de/GovOPlaN/govoplan-addresses/commit/d4fa024034e52924e9573b13a46dd0cee2287d9e) - govoplan-calendar: [v0.1.24](https://git.add-ideas.de/GovOPlaN/govoplan-calendar/src/tag/v0.1.24) · [8e36f8b3d203](https://git.add-ideas.de/GovOPlaN/govoplan-calendar/commit/8e36f8b3d203fc0b08ea0487540fb7ac597cb8c9) - govoplan-campaign: [v0.1.29](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/src/tag/v0.1.29) · [19437ce37855](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/commit/19437ce378553f6ab559e3c0cba44150e495ef52) - govoplan-cases: [v0.1.25](https://git.add-ideas.de/GovOPlaN/govoplan-cases/src/tag/v0.1.25) · [0337e0cf0b37](https://git.add-ideas.de/GovOPlaN/govoplan-cases/commit/0337e0cf0b37a9b89dd51cccf28500649361fb37) - govoplan-committee: [v0.1.22](https://git.add-ideas.de/GovOPlaN/govoplan-committee/src/tag/v0.1.22) · [96c48a7f2d3a](https://git.add-ideas.de/GovOPlaN/govoplan-committee/commit/96c48a7f2d3a8fa36f44d5806a14003e40cc4807) - govoplan-connectors: [v0.1.27](https://git.add-ideas.de/GovOPlaN/govoplan-connectors/src/tag/v0.1.27) · [889cafaf207a](https://git.add-ideas.de/GovOPlaN/govoplan-connectors/commit/889cafaf207a2e7e495c64ccbac07d9faf5a5dce) - govoplan-core: [v0.1.46](https://git.add-ideas.de/GovOPlaN/govoplan-core/src/tag/v0.1.46) · [6591aaa3fdaf](https://git.add-ideas.de/GovOPlaN/govoplan-core/commit/6591aaa3fdaf19d1900ccff326c56a0b0969d9fa) - govoplan-dataflow: [v0.1.25](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/src/tag/v0.1.25) · [a6c5bab3a4b7](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/commit/a6c5bab3a4b7a52cb0f9b07911216ce61f362bbe) - govoplan-datasources: [v0.1.26](https://git.add-ideas.de/GovOPlaN/govoplan-datasources/src/tag/v0.1.26) · [9d067f1baded](https://git.add-ideas.de/GovOPlaN/govoplan-datasources/commit/9d067f1baded6e65a221fbba0b46fdd953404b51) - govoplan-files: [v0.1.27](https://git.add-ideas.de/GovOPlaN/govoplan-files/src/tag/v0.1.27) · [61625fb00fed](https://git.add-ideas.de/GovOPlaN/govoplan-files/commit/61625fb00fedbd4474aab1854ba9cedaf5fd18f3) - govoplan-forms-runtime: [v0.1.22](https://git.add-ideas.de/GovOPlaN/govoplan-forms-runtime/src/tag/v0.1.22) · [1236b861b812](https://git.add-ideas.de/GovOPlaN/govoplan-forms-runtime/commit/1236b861b8128b9fa1ce30c51a32856554d8c3df) - govoplan-idm: [v0.1.26](https://git.add-ideas.de/GovOPlaN/govoplan-idm/src/tag/v0.1.26) · [ba04593e2914](https://git.add-ideas.de/GovOPlaN/govoplan-idm/commit/ba04593e29140298687a29a6bdf46d1376974c02) - govoplan-mail: [v0.1.28](https://git.add-ideas.de/GovOPlaN/govoplan-mail/src/tag/v0.1.28) · [d9262982147f](https://git.add-ideas.de/GovOPlaN/govoplan-mail/commit/d9262982147ff70b4edbc8990d0bf8ccb5f190a8) - govoplan-reporting: [v0.1.22](https://git.add-ideas.de/GovOPlaN/govoplan-reporting/src/tag/v0.1.22) · [8253bd0dd6fe](https://git.add-ideas.de/GovOPlaN/govoplan-reporting/commit/8253bd0dd6fe93b739b7482a04c058f2bed3aff4) - govoplan-tickets: [v0.1.23](https://git.add-ideas.de/GovOPlaN/govoplan-tickets/src/tag/v0.1.23) · [2be4a0c598a3](https://git.add-ideas.de/GovOPlaN/govoplan-tickets/commit/2be4a0c598a32a5ce044320f2a1311a192b9d5e3) Remote main branches and annotated tags were verified against the exact coordinated commits. The implementation-specific tests and EN/DE documentation recorded above remain the acceptance evidence. Cross-module focused checks, manifest shapes, 63 WebUI build configurations and 230 browser tests passed. This records source publication, not a package-registry release, runtime image, website deployment or production rollout. Database correction to the previous local-only note: the already-running development server had automatically applied Files a2b3c4d5e701, Mail b5d6e7f8091a, Connectors d2a4c6e8f0b1 and Datasources e2b8d4a0f6c3 during reloads. Their heads and expected schema/indexes were verified in the local development PostgreSQL database. A current-state backup was captured and restored into a private, isolated PostgreSQL 16 cluster; this was not a pre-upgrade backup. The supported no-op migration preserved all 281 tables and 142,287 rows by per-row SHA-256 comparison, and retained the same migration heads. Files source identity recomputation matched all 7,385 rows (2 non-null identities, zero mismatches). Both Datasources PostgreSQL concurrency tests, previously skipped without a dedicated database, passed against that isolated cluster. The test cluster was stopped. No real mail delivery, POP3 retrieval/deletion, legacy mailbox reconciliation or external connector import was initiated. Representative production load/deadline testing and real-provider race/compatibility verification remain separate operational rollout checks; this is not security certification. Operational follow-up: https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/299. Closing this implementation issue: its code, regression, documentation and source-publication criteria are satisfied. Operational checks above are not claimed complete. <!-- govoplan-integrity-publication-20260908 -->
zemion removed the
status
in-progress
 label 2026-09-08 12:57:28 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/api
HTTP API contracts, routers, schemas, or API smoke behavior.
 area/auth
Authentication, sessions, access bootstrap, or login behavior.
 area/db
Database sessions, models, transactions, or persistence primitives.
 area/devex
Local developer workflow, scripts, tests, tooling, or release helpers.
 area/docs
Durable documentation and project guidance.
 area/governance
Governance policy, audit, privacy, retention, or compliance behavior.
 area/marketing
Public website, product messaging, publication copy, or legal page content.
 area/migrations
Alembic migrations, schema bootstrap, or persistence evolution.
 area/module-system
Module discovery, manifests, capabilities, routing, or optional integrations.
 area/rbac
Permissions, roles, delegation, or authorization policy.
 area/release
Versioning, release locks, tags, packaging, or dependency pins.
 area/security
Security posture, static analysis, supply-chain hardening, or vulnerability remediation.
 area/tenancy
Tenant boundaries, provisioning, or tenant-scoped data behavior.
 area/webui
Shared WebUI shell, frontend components, routing, or frontend tests.
 audit/complexity
Complexity finding from Radon, Xenon, or equivalent maintainability scans.
 audit/duplication
Duplicated-code finding from jscpd or equivalent similarity scans.
 audit/false-positive
Audit finding reviewed as a narrow false positive or acceptable risk.
 audit/needs-design
Audit finding that needs an architectural or product decision before implementation.
 audit/quick-fix
Audit finding that appears narrow and directly fixable.
 audit/structural
Audit finding that needs design, refactoring, or behavior review.
 codex/needs-human
Needs an explicit human decision before Codex should implement.
 codex/ready
Suitable for Codex to pick up with the existing issue context.
 module/access
GovOPlaN access, identity, authentication, RBAC, and administration behavior.
 module/addresses
GovOPlaN Addresses module behavior or integration.
 module/admin
GovOPlaN Admin module behavior or integration.
 module/appointments
GovOPlaN Appointments module behavior or integration.
 module/approvals
GovOPlaN Approvals module behavior or integration.
 module/audit
GovOPlaN Audit module behavior or integration.
 module/calendar
GovOPlaN Calendar module behavior or integration.
 module/campaign
GovOPlaN campaign module behavior or integration.
 module/cases
GovOPlaN Cases module behavior or integration.
 module/committee
GovOPlaN Committee module behavior or integration.
 module/connectors
GovOPlaN Connectors module behavior or integration.
 module/core
GovOPlaN core runner, shared primitives, shell, or extension points.
 module/dashboard
GovOPlaN Dashboard module behavior or integration.
 module/dataflow
GovOPlaN Dataflow module behavior or integration.
 module/datasources
GovOPlaN governed datasource contracts, catalogs, and integrations.
 module/decisions
GovOPlaN formal Decisions module behavior or integration.
 module/dist-lists
GovOPlaN Distribution Lists module behavior or integration.
 module/dms
GovOPlaN Dms module behavior or integration.
 module/docs
GovOPlaN Docs module behavior or integration.
 module/encryption
GovOPlaN Encryption key custody, cryptographic policy, and E2EE integration.
 module/erp
GovOPlaN Erp module behavior or integration.
 module/evaluation
GovOPlaN Evaluation module behavior or integration.
 module/files
GovOPlaN files module behavior or integration.
 module/fit-connect
GovOPlaN Fit Connect module behavior or integration.
 module/forms
GovOPlaN Forms module behavior or integration.
 module/forms-runtime
GovOPlaN Forms Runtime module behavior or integration.
 module/helpdesk
GovOPlaN Helpdesk module behavior or integration.
 module/identity
GovOPlaN Identity module behavior or integration.
 module/identity-trust
GovOPlaN Identity Trust module behavior or integration.
 module/idm
GovOPlaN Idm module behavior or integration.
 module/ledger
GovOPlaN Ledger module behavior or integration.
 module/mail
GovOPlaN mail module behavior or integration.
 module/mandates
GovOPlaN Mandates, jurisdiction, responsibility, and authority behavior or integration.
 module/notifications
GovOPlaN Notifications module behavior or integration.
 module/ops
GovOPlaN Ops module behavior or integration.
 module/organizations
GovOPlaN Organizations module behavior or integration.
 module/parties
GovOPlaN procedure Parties, representation, and delivery-authority behavior or integration.
 module/payments
GovOPlaN Payments module behavior or integration.
 module/permits
GovOPlaN Permits module behavior or integration.
 module/policy
GovOPlaN Policy module behavior or integration.
 module/poll
GovOPlaN Poll module behavior or integration.
 module/portal
GovOPlaN Portal module behavior or integration.
 module/postbox
GovOPlaN Postbox module behavior or integration.
 module/projects
GovOPlaN Projects module behavior or integration.
 module/quick-access
GovOPlaN configurable task-local Quick Access behavior and integrations.
 module/records
GovOPlaN Records and eAkte lifecycle behavior or integration.
 module/reporting
GovOPlaN Reporting module behavior or integration.
 module/risk-compliance
GovOPlaN Risk Compliance module behavior or integration.
 module/scheduling
GovOPlaN Scheduling module behavior or integration.
 module/search
GovOPlaN Search module behavior or integration.
 module/services
GovOPlaN versioned institutional Services behavior or integration.
 module/tasks
GovOPlaN Tasks module behavior or integration.
 module/templates
GovOPlaN Templates module behavior or integration.
 module/tenancy
GovOPlaN Tenancy module behavior or integration.
 module/tickets
GovOPlaN Tickets module behavior or integration.
 module/views
GovOPlaN governed task views, interface projections, and workflow view integration.
 module/voting
GovOPlaN Voting module behavior or integration.
 module/wiki
GovOPlaN Wiki module behavior or integration.
 module/workflow
GovOPlaN Workflow module behavior or integration.
 module/workflow-engine
GovOPlaN Workflow Engine runtime, persistence, or integration.
 module/xoev
GovOPlaN Xoev module behavior or integration.
 module/xrechnung
GovOPlaN Xrechnung module behavior or integration.
 module/xta-osci
GovOPlaN Xta Osci module behavior or integration.
priority
p0
Immediate stop-the-line priority.
priority
p1
High priority for the next focused work window.
priority
p2
Normal planned priority.
priority
p3
Low priority or opportunistic cleanup.
source/backlog-import
Imported from markdown backlog, roadmap, plan, or TODO files.
 source/security-audit
Created from a structured security or code-quality audit report.
 source/todo-scan
Imported from inline TODO/FIXME/HACK markers by the Gitea TODO importer.
status
blocked
Cannot progress without a decision, dependency, credential, or external change.
status
in-progress
Currently being worked.
status
needs-info
Needs clarifying input before implementation can proceed safely.
status
ready
Ready for implementation.
status
triage
Needs review, ownership, priority, or acceptance criteria.
type
bug
A reproducible defect, regression, or incorrect behavior.
type
debt
Cleanup, refactoring, risk reduction, or deferred engineering work.
type
docs
Documentation, process, or developer workflow work.
type
feature
New user-visible behavior or platform capability.
type
task
Implementation, maintenance, migration, or operational work.
type
user-story
End-to-end user journey or real-world process story used to steer product slices.
No labels
priority
p1
type
bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
govoplan-package-publisher zemion (Albrecht Degering)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-core#298