[Feature] Platform-wide conditional GET and delta API foundation #222

New Issue

zemion · 2026-07-09T11:32:59+02:00

zemion commented

2026-07-09 11:32:59 +02:00

Problem

Repeated reloads fetch complete JSON payloads even when the user-visible data has not changed. This is especially expensive for table/list views and settings pages where one row or one setting changes but the frontend reloads the whole resource.

Proposed Capability

Add a platform-wide conditional GET and delta foundation:

GET endpoints can return scoped private ETags and answer matching If-None-Match requests with 304 Not Modified.
The shared WebUI API client keeps an in-memory conditional response cache, sends If-None-Match for reusable GET requests, and returns the cached payload on 304.
The platform defines a reusable delta response contract for collection endpoints that can support row-level changes later without inventing per-module formats.
Unsafe writes invalidate the frontend conditional cache generation.

Ownership

Owning repository: govoplan-core
Related module repositories: all modules with table/list endpoints, starting with campaign, files, mail, calendar, docs, ops
Extension point or integration boundary: shared FastAPI middleware/helpers and shared WebUI API client

Acceptance Criteria

JSON GET responses receive private ETags when safe to cache conditionally.
If-None-Match with a matching current ETag returns 304 without a response body.
The shared WebUI API client sends If-None-Match for cached reusable GETs and returns cached payloads for 304 responses.
Unsafe API methods invalidate the frontend conditional cache.
Delta collection response types are documented and exported for module endpoints that can expose revision/watermark based updates.
Tests cover ETag generation, 304 behavior, auth/tenant scoped variation, frontend 304 handling, and write invalidation.

Verification Target

Backend unit/API tests for conditional GET behavior.
WebUI TypeScript build and focused client tests.
A manual reload of admin/settings/table views should show smaller network responses after the first load when data has not changed.

## Problem Repeated reloads fetch complete JSON payloads even when the user-visible data has not changed. This is especially expensive for table/list views and settings pages where one row or one setting changes but the frontend reloads the whole resource. ## Proposed Capability Add a platform-wide conditional GET and delta foundation: - GET endpoints can return scoped private ETags and answer matching If-None-Match requests with 304 Not Modified. - The shared WebUI API client keeps an in-memory conditional response cache, sends If-None-Match for reusable GET requests, and returns the cached payload on 304. - The platform defines a reusable delta response contract for collection endpoints that can support row-level changes later without inventing per-module formats. - Unsafe writes invalidate the frontend conditional cache generation. ## Ownership - Owning repository: govoplan-core - Related module repositories: all modules with table/list endpoints, starting with campaign, files, mail, calendar, docs, ops - Extension point or integration boundary: shared FastAPI middleware/helpers and shared WebUI API client ## Acceptance Criteria - [ ] JSON GET responses receive private ETags when safe to cache conditionally. - [ ] If-None-Match with a matching current ETag returns 304 without a response body. - [ ] The shared WebUI API client sends If-None-Match for cached reusable GETs and returns cached payloads for 304 responses. - [ ] Unsafe API methods invalidate the frontend conditional cache. - [ ] Delta collection response types are documented and exported for module endpoints that can expose revision/watermark based updates. - [ ] Tests cover ETag generation, 304 behavior, auth/tenant scoped variation, frontend 304 handling, and write invalidation. ## Verification Target - Backend unit/API tests for conditional GET behavior. - WebUI TypeScript build and focused client tests. - A manual reload of admin/settings/table views should show smaller network responses after the first load when data has not changed.

zemion added the

module/core

codex/ready

labels 2026-07-09 11:32:59 +02:00

zemion commented

2026-07-09 11:38:51 +02:00

Implemented the first platform foundation pass in core:

Added central conditional JSON GET middleware for successful JSON GET responses.
Middleware emits private weak ETags, honors If-None-Match, returns 304 for unchanged payloads, and skips cookie, encoded, download, non-JSON, no-store, and non-200 responses.
Updated the shared WebUI apiFetch client to keep an in-memory conditional cache, send If-None-Match for reusable GETs, return cached payloads on 304, and clear cache generations after unsafe methods.
Added shared backend and frontend delta collection DTOs plus docs/API_CONDITIONAL_DELTA.md for module endpoints that can expose row-level deltas later.

Verification passed:

./.venv/bin/python -m unittest tests.test_conditional_requests
./.venv/bin/python -m unittest tests.test_core_events
./.venv/bin/python -m unittest tests.test_api_smoke.ApiSmokeTests.test_recipient_import_mapping_profiles_are_db_backed
npm run build

Implemented the first platform foundation pass in core: - Added central conditional JSON GET middleware for successful JSON GET responses. - Middleware emits private weak ETags, honors If-None-Match, returns 304 for unchanged payloads, and skips cookie, encoded, download, non-JSON, no-store, and non-200 responses. - Updated the shared WebUI apiFetch client to keep an in-memory conditional cache, send If-None-Match for reusable GETs, return cached payloads on 304, and clear cache generations after unsafe methods. - Added shared backend and frontend delta collection DTOs plus docs/API_CONDITIONAL_DELTA.md for module endpoints that can expose row-level deltas later. Verification passed: - ./.venv/bin/python -m unittest tests.test_conditional_requests - ./.venv/bin/python -m unittest tests.test_core_events - ./.venv/bin/python -m unittest tests.test_api_smoke.ApiSmokeTests.test_recipient_import_mapping_profiles_are_db_backed - npm run build

zemion commented

2026-07-09 11:41:14 +02:00

Follow-up verification added:

Extended tests.test_conditional_requests with representative endpoint coverage for /api/v1/platform/status and /api/v1/platform/modules.
The test verifies both endpoints emit ETags and return 304 with an empty body for matching If-None-Match.

Verification rerun:

./.venv/bin/python -m unittest tests.test_conditional_requests
npm run build

Follow-up verification added: - Extended tests.test_conditional_requests with representative endpoint coverage for /api/v1/platform/status and /api/v1/platform/modules. - The test verifies both endpoints emit ETags and return 304 with an empty body for matching If-None-Match. Verification rerun: - ./.venv/bin/python -m unittest tests.test_conditional_requests - npm run build

zemion commented

2026-07-09 12:04:02 +02:00

Codex State: progress

Summary

Implemented core-wide monotonic change sequence with seq: watermarks and Alembic head.
Added files module change tracking for file assets, folders, and share visibility changes.
Added GET /api/v1/files/delta plus WebUI API/capability types; full snapshot without since, incremental response with tombstones when since is provided.

Changed Files

src/govoplan_core/core/change_sequence.py
alembic/versions/3f4a5b6c7d8e_core_change_sequence.py
src/govoplan_files/backend/change_tracking.py
src/govoplan_files/backend/router.py
webui/src/types.ts

Verification

python -m unittest tests.test_change_sequence tests.test_conditional_requests tests.test_api_smoke.ApiSmokeTests.test_managed_file_runtime_flow tests.test_api_smoke.ApiSmokeTests.test_files_delta_tracks_uploads_folders_and_delete_tombstones
python -m unittest tests.test_database_migrations.DatabaseMigrationTests.test_repairs_create_all_schema_drift_and_upgrades_to_head tests.test_database_migrations.DatabaseMigrationTests.test_repairs_current_schema_stamped_at_file_folders tests.test_module_system.ModuleSystemTests.test_module_migrations_are_registered_only_for_enabled_modules
npm run build
npm run test:module-capabilities

Next / Blocked

Campaign workspace data remains the next high-value retrofit target.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Implemented core-wide monotonic change sequence with seq: watermarks and Alembic head. - Added files module change tracking for file assets, folders, and share visibility changes. - Added GET /api/v1/files/delta plus WebUI API/capability types; full snapshot without since, incremental response with tombstones when since is provided. ### Changed Files - `src/govoplan_core/core/change_sequence.py` - `alembic/versions/3f4a5b6c7d8e_core_change_sequence.py` - `src/govoplan_files/backend/change_tracking.py` - `src/govoplan_files/backend/router.py` - `webui/src/types.ts` ### Verification - `python -m unittest tests.test_change_sequence tests.test_conditional_requests tests.test_api_smoke.ApiSmokeTests.test_managed_file_runtime_flow tests.test_api_smoke.ApiSmokeTests.test_files_delta_tracks_uploads_folders_and_delete_tombstones` - `python -m unittest tests.test_database_migrations.DatabaseMigrationTests.test_repairs_create_all_schema_drift_and_upgrades_to_head tests.test_database_migrations.DatabaseMigrationTests.test_repairs_current_schema_stamped_at_file_folders tests.test_module_system.ModuleSystemTests.test_module_migrations_are_registered_only_for_enabled_modules` - `npm run build` - `npm run test:module-capabilities` ### Next / Blocked - Campaign workspace data remains the next high-value retrofit target. Suggested status label: `status/in-progress`

zemion commented

2026-07-09 12:06:21 +02:00

Codex State: progress

Summary

Updated the files WebUI to consume GET /api/v1/files/delta for managed space reloads.
Managed file chooser now uses the delta full-snapshot response instead of separate files/folders GETs.

Changed Files

webui/src/features/files/FilesPage.tsx
webui/src/features/files/components/ManagedFileChooser.tsx

Verification

npm run build
npm run test:module-capabilities

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Updated the files WebUI to consume GET /api/v1/files/delta for managed space reloads. - Managed file chooser now uses the delta full-snapshot response instead of separate files/folders GETs. ### Changed Files - `webui/src/features/files/FilesPage.tsx` - `webui/src/features/files/components/ManagedFileChooser.tsx` ### Verification - `npm run build` - `npm run test:module-capabilities` Suggested status label: `status/in-progress`

zemion commented

2026-07-09 12:27:45 +02:00

Codex State: progress

Summary

Files delta privacy regression added; broad readers no longer receive private tombstones for another user's deleted files.
Campaign is now a delta consumer: campaign list and campaign workspace delta endpoints use the shared core_change_sequence contract and stale-watermark full snapshot fallback.
Campaign WebUI list and workspace hook now consume the delta endpoints with in-memory merge/delete handling.
Retention groundwork added through sequence_watermark_is_expired and documented retained-window semantics.

Changed Files

src/govoplan_core/core/change_sequence.py
docs/MODULE_ARCHITECTURE.md
tests/test_change_sequence.py
tests/test_api_smoke.py
../govoplan-files/src/govoplan_files/backend/router.py
../govoplan-campaign/src/govoplan_campaign/backend/change_tracking.py
../govoplan-campaign/src/govoplan_campaign/backend/router.py
../govoplan-campaign/src/govoplan_campaign/backend/schemas.py
../govoplan-campaign/webui/src/api/campaigns.ts
../govoplan-campaign/webui/src/features/campaigns/CampaignListPage.tsx
../govoplan-campaign/webui/src/features/campaigns/hooks/useCampaignWorkspaceData.ts

Verification

./.venv/bin/python -m unittest tests.test_change_sequence tests.test_api_smoke.ApiSmokeTests.test_files_delta_omits_private_tombstones_from_broad_reader_feed tests.test_api_smoke.ApiSmokeTests.test_files_delta_tracks_uploads_folders_and_delete_tombstones tests.test_api_smoke.ApiSmokeTests.test_campaign_delta_tracks_create_update_and_delete_tombstone tests.test_api_smoke.ApiSmokeTests.test_campaign_workspace_delta_tracks_version_autosave
./.venv/bin/python -m py_compile src/govoplan_core/core/change_sequence.py ../govoplan-files/src/govoplan_files/backend/router.py ../govoplan-campaign/src/govoplan_campaign/backend/change_tracking.py ../govoplan-campaign/src/govoplan_campaign/backend/router.py ../govoplan-campaign/src/govoplan_campaign/backend/schemas.py
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:/home/zemion/.nvm/versions/node/v22.22.3/bin:/home/zemion/.local/bin:/home/zemion/.codex/tmp/arg0/codex-arg0WEB8kZ:/app/bin:/app/bin:/app/bin:/usr/bin:/usr/lib/sdk/openjdk25/bin:/home/zemion/.var/app/com.vscodium.codium/data/node/bin:/home/zemion/.var/app/com.vscodium.codium/data/cargo/bin:/home/zemion/.var/app/com.vscodium.codium/data/python/bin:/home/zemion/.var/app/com.vscodium.codium/data/codium/extensions/openai.chatgpt-26.5623.101652-linux-x64/bin/linux-x86_64 npm run build (govoplan-core/webui)
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:/home/zemion/.nvm/versions/node/v22.22.3/bin:/home/zemion/.local/bin:/home/zemion/.codex/tmp/arg0/codex-arg0WEB8kZ:/app/bin:/app/bin:/app/bin:/usr/bin:/usr/lib/sdk/openjdk25/bin:/home/zemion/.var/app/com.vscodium.codium/data/node/bin:/home/zemion/.var/app/com.vscodium.codium/data/cargo/bin:/home/zemion/.var/app/com.vscodium.codium/data/python/bin:/home/zemion/.var/app/com.vscodium.codium/data/codium/extensions/openai.chatgpt-26.5623.101652-linux-x64/bin/linux-x86_64 npm run test:module-capabilities (govoplan-core/webui)
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:/home/zemion/.nvm/versions/node/v22.22.3/bin:/home/zemion/.local/bin:/home/zemion/.codex/tmp/arg0/codex-arg0WEB8kZ:/app/bin:/app/bin:/app/bin:/usr/bin:/usr/lib/sdk/openjdk25/bin:/home/zemion/.var/app/com.vscodium.codium/data/node/bin:/home/zemion/.var/app/com.vscodium.codium/data/cargo/bin:/home/zemion/.var/app/com.vscodium.codium/data/python/bin:/home/zemion/.var/app/com.vscodium.codium/data/codium/extensions/openai.chatgpt-26.5623.101652-linux-x64/bin/linux-x86_64 npm run test:policy-ui (govoplan-campaign/webui)

Next / Blocked

Next retrofit target should be the remaining high-volume campaign job/report table feeds, followed by reusable helpers for module delta merge hooks.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Files delta privacy regression added; broad readers no longer receive private tombstones for another user's deleted files. - Campaign is now a delta consumer: campaign list and campaign workspace delta endpoints use the shared core_change_sequence contract and stale-watermark full snapshot fallback. - Campaign WebUI list and workspace hook now consume the delta endpoints with in-memory merge/delete handling. - Retention groundwork added through sequence_watermark_is_expired and documented retained-window semantics. ### Changed Files - `src/govoplan_core/core/change_sequence.py` - `docs/MODULE_ARCHITECTURE.md` - `tests/test_change_sequence.py` - `tests/test_api_smoke.py` - `../govoplan-files/src/govoplan_files/backend/router.py` - `../govoplan-campaign/src/govoplan_campaign/backend/change_tracking.py` - `../govoplan-campaign/src/govoplan_campaign/backend/router.py` - `../govoplan-campaign/src/govoplan_campaign/backend/schemas.py` - `../govoplan-campaign/webui/src/api/campaigns.ts` - `../govoplan-campaign/webui/src/features/campaigns/CampaignListPage.tsx` - `../govoplan-campaign/webui/src/features/campaigns/hooks/useCampaignWorkspaceData.ts` ### Verification - `./.venv/bin/python -m unittest tests.test_change_sequence tests.test_api_smoke.ApiSmokeTests.test_files_delta_omits_private_tombstones_from_broad_reader_feed tests.test_api_smoke.ApiSmokeTests.test_files_delta_tracks_uploads_folders_and_delete_tombstones tests.test_api_smoke.ApiSmokeTests.test_campaign_delta_tracks_create_update_and_delete_tombstone tests.test_api_smoke.ApiSmokeTests.test_campaign_workspace_delta_tracks_version_autosave` - `./.venv/bin/python -m py_compile src/govoplan_core/core/change_sequence.py ../govoplan-files/src/govoplan_files/backend/router.py ../govoplan-campaign/src/govoplan_campaign/backend/change_tracking.py ../govoplan-campaign/src/govoplan_campaign/backend/router.py ../govoplan-campaign/src/govoplan_campaign/backend/schemas.py` - `PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:/home/zemion/.nvm/versions/node/v22.22.3/bin:/home/zemion/.local/bin:/home/zemion/.codex/tmp/arg0/codex-arg0WEB8kZ:/app/bin:/app/bin:/app/bin:/usr/bin:/usr/lib/sdk/openjdk25/bin:/home/zemion/.var/app/com.vscodium.codium/data/node/bin:/home/zemion/.var/app/com.vscodium.codium/data/cargo/bin:/home/zemion/.var/app/com.vscodium.codium/data/python/bin:/home/zemion/.var/app/com.vscodium.codium/data/codium/extensions/openai.chatgpt-26.5623.101652-linux-x64/bin/linux-x86_64 npm run build (govoplan-core/webui)` - `PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:/home/zemion/.nvm/versions/node/v22.22.3/bin:/home/zemion/.local/bin:/home/zemion/.codex/tmp/arg0/codex-arg0WEB8kZ:/app/bin:/app/bin:/app/bin:/usr/bin:/usr/lib/sdk/openjdk25/bin:/home/zemion/.var/app/com.vscodium.codium/data/node/bin:/home/zemion/.var/app/com.vscodium.codium/data/cargo/bin:/home/zemion/.var/app/com.vscodium.codium/data/python/bin:/home/zemion/.var/app/com.vscodium.codium/data/codium/extensions/openai.chatgpt-26.5623.101652-linux-x64/bin/linux-x86_64 npm run test:module-capabilities (govoplan-core/webui)` - `PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:/home/zemion/.nvm/versions/node/v22.22.3/bin:/home/zemion/.local/bin:/home/zemion/.codex/tmp/arg0/codex-arg0WEB8kZ:/app/bin:/app/bin:/app/bin:/usr/bin:/usr/lib/sdk/openjdk25/bin:/home/zemion/.var/app/com.vscodium.codium/data/node/bin:/home/zemion/.var/app/com.vscodium.codium/data/cargo/bin:/home/zemion/.var/app/com.vscodium.codium/data/python/bin:/home/zemion/.var/app/com.vscodium.codium/data/codium/extensions/openai.chatgpt-26.5623.101652-linux-x64/bin/linux-x86_64 npm run test:policy-ui (govoplan-campaign/webui)` ### Next / Blocked - Next retrofit target should be the remaining high-volume campaign job/report table feeds, followed by reusable helpers for module delta merge hooks. Suggested status label: `status/in-progress`

zemion commented

2026-07-09 12:39:44 +02:00

Codex State: progress

Summary

Campaign job/report table delta retrofit is implemented: GET /api/v1/campaigns/{campaign_id}/jobs/delta now supports first full page snapshots and stable row deltas for unfiltered job updates.
Filtered job views, created/deleted jobs, and stale watermarks intentionally fall back to full page snapshots to preserve pagination correctness.
Campaign report page now keeps a per-query job watermark and merges job row deltas through a shared core WebUI helper.
Core WebUI now exports mergeDeltaRows/deltaDeletedIds for future module delta consumers.

Changed Files

webui/src/utils/delta.ts
webui/src/index.ts
docs/MODULE_ARCHITECTURE.md
tests/test_api_smoke.py
../govoplan-campaign/src/govoplan_campaign/backend/router.py
../govoplan-campaign/src/govoplan_campaign/backend/schemas.py
../govoplan-campaign/webui/src/api/campaigns.ts
../govoplan-campaign/webui/src/features/campaigns/CampaignListPage.tsx
../govoplan-campaign/webui/src/features/campaigns/CampaignReportPage.tsx

Verification

./.venv/bin/python -m unittest tests.test_change_sequence tests.test_api_smoke.ApiSmokeTests.test_campaign_jobs_delta_tracks_single_job_update tests.test_api_smoke.ApiSmokeTests.test_campaign_delta_tracks_create_update_and_delete_tombstone tests.test_api_smoke.ApiSmokeTests.test_campaign_workspace_delta_tracks_version_autosave tests.test_api_smoke.ApiSmokeTests.test_files_delta_omits_private_tombstones_from_broad_reader_feed
./.venv/bin/python -m py_compile src/govoplan_core/core/change_sequence.py ../govoplan-campaign/src/govoplan_campaign/backend/change_tracking.py ../govoplan-campaign/src/govoplan_campaign/backend/router.py ../govoplan-campaign/src/govoplan_campaign/backend/schemas.py ../govoplan-files/src/govoplan_files/backend/router.py
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:/home/zemion/.nvm/versions/node/v22.22.3/bin:/home/zemion/.local/bin:/home/zemion/.codex/tmp/arg0/codex-arg0WEB8kZ:/app/bin:/app/bin:/app/bin:/usr/bin:/usr/lib/sdk/openjdk25/bin:/home/zemion/.var/app/com.vscodium.codium/data/node/bin:/home/zemion/.var/app/com.vscodium.codium/data/cargo/bin:/home/zemion/.var/app/com.vscodium.codium/data/python/bin:/home/zemion/.var/app/com.vscodium.codium/data/codium/extensions/openai.chatgpt-26.5623.101652-linux-x64/bin/linux-x86_64 npm run build (govoplan-core/webui)
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:/home/zemion/.nvm/versions/node/v22.22.3/bin:/home/zemion/.local/bin:/home/zemion/.codex/tmp/arg0/codex-arg0WEB8kZ:/app/bin:/app/bin:/app/bin:/usr/bin:/usr/lib/sdk/openjdk25/bin:/home/zemion/.var/app/com.vscodium.codium/data/node/bin:/home/zemion/.var/app/com.vscodium.codium/data/cargo/bin:/home/zemion/.var/app/com.vscodium.codium/data/python/bin:/home/zemion/.var/app/com.vscodium.codium/data/codium/extensions/openai.chatgpt-26.5623.101652-linux-x64/bin/linux-x86_64 npm run test:module-capabilities (govoplan-core/webui)
PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:/home/zemion/.nvm/versions/node/v22.22.3/bin:/home/zemion/.local/bin:/home/zemion/.codex/tmp/arg0/codex-arg0WEB8kZ:/app/bin:/app/bin:/app/bin:/usr/bin:/usr/lib/sdk/openjdk25/bin:/home/zemion/.var/app/com.vscodium.codium/data/node/bin:/home/zemion/.var/app/com.vscodium.codium/data/cargo/bin:/home/zemion/.var/app/com.vscodium.codium/data/python/bin:/home/zemion/.var/app/com.vscodium.codium/data/codium/extensions/openai.chatgpt-26.5623.101652-linux-x64/bin/linux-x86_64 npm run test:policy-ui (govoplan-campaign/webui)

Next / Blocked

Next logical delta consumers: ReviewSendPage inspection jobs and operator queue summaries, then a small reusable React hook for per-query watermarks.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Campaign job/report table delta retrofit is implemented: GET /api/v1/campaigns/{campaign_id}/jobs/delta now supports first full page snapshots and stable row deltas for unfiltered job updates. - Filtered job views, created/deleted jobs, and stale watermarks intentionally fall back to full page snapshots to preserve pagination correctness. - Campaign report page now keeps a per-query job watermark and merges job row deltas through a shared core WebUI helper. - Core WebUI now exports mergeDeltaRows/deltaDeletedIds for future module delta consumers. ### Changed Files - `webui/src/utils/delta.ts` - `webui/src/index.ts` - `docs/MODULE_ARCHITECTURE.md` - `tests/test_api_smoke.py` - `../govoplan-campaign/src/govoplan_campaign/backend/router.py` - `../govoplan-campaign/src/govoplan_campaign/backend/schemas.py` - `../govoplan-campaign/webui/src/api/campaigns.ts` - `../govoplan-campaign/webui/src/features/campaigns/CampaignListPage.tsx` - `../govoplan-campaign/webui/src/features/campaigns/CampaignReportPage.tsx` ### Verification - `./.venv/bin/python -m unittest tests.test_change_sequence tests.test_api_smoke.ApiSmokeTests.test_campaign_jobs_delta_tracks_single_job_update tests.test_api_smoke.ApiSmokeTests.test_campaign_delta_tracks_create_update_and_delete_tombstone tests.test_api_smoke.ApiSmokeTests.test_campaign_workspace_delta_tracks_version_autosave tests.test_api_smoke.ApiSmokeTests.test_files_delta_omits_private_tombstones_from_broad_reader_feed` - `./.venv/bin/python -m py_compile src/govoplan_core/core/change_sequence.py ../govoplan-campaign/src/govoplan_campaign/backend/change_tracking.py ../govoplan-campaign/src/govoplan_campaign/backend/router.py ../govoplan-campaign/src/govoplan_campaign/backend/schemas.py ../govoplan-files/src/govoplan_files/backend/router.py` - `PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:/home/zemion/.nvm/versions/node/v22.22.3/bin:/home/zemion/.local/bin:/home/zemion/.codex/tmp/arg0/codex-arg0WEB8kZ:/app/bin:/app/bin:/app/bin:/usr/bin:/usr/lib/sdk/openjdk25/bin:/home/zemion/.var/app/com.vscodium.codium/data/node/bin:/home/zemion/.var/app/com.vscodium.codium/data/cargo/bin:/home/zemion/.var/app/com.vscodium.codium/data/python/bin:/home/zemion/.var/app/com.vscodium.codium/data/codium/extensions/openai.chatgpt-26.5623.101652-linux-x64/bin/linux-x86_64 npm run build (govoplan-core/webui)` - `PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:/home/zemion/.nvm/versions/node/v22.22.3/bin:/home/zemion/.local/bin:/home/zemion/.codex/tmp/arg0/codex-arg0WEB8kZ:/app/bin:/app/bin:/app/bin:/usr/bin:/usr/lib/sdk/openjdk25/bin:/home/zemion/.var/app/com.vscodium.codium/data/node/bin:/home/zemion/.var/app/com.vscodium.codium/data/cargo/bin:/home/zemion/.var/app/com.vscodium.codium/data/python/bin:/home/zemion/.var/app/com.vscodium.codium/data/codium/extensions/openai.chatgpt-26.5623.101652-linux-x64/bin/linux-x86_64 npm run test:module-capabilities (govoplan-core/webui)` - `PATH=/mnt/DATA/git/govoplan-core/webui/node_modules/.bin:/home/zemion/.nvm/versions/node/v22.22.3/bin:/home/zemion/.local/bin:/home/zemion/.codex/tmp/arg0/codex-arg0WEB8kZ:/app/bin:/app/bin:/app/bin:/usr/bin:/usr/lib/sdk/openjdk25/bin:/home/zemion/.var/app/com.vscodium.codium/data/node/bin:/home/zemion/.var/app/com.vscodium.codium/data/cargo/bin:/home/zemion/.var/app/com.vscodium.codium/data/python/bin:/home/zemion/.var/app/com.vscodium.codium/data/codium/extensions/openai.chatgpt-26.5623.101652-linux-x64/bin/linux-x86_64 npm run test:policy-ui (govoplan-campaign/webui)` ### Next / Blocked - Next logical delta consumers: ReviewSendPage inspection jobs and operator queue summaries, then a small reusable React hook for per-query watermarks. Suggested status label: `status/in-progress`

zemion commented

2026-07-09 13:04:22 +02:00

Codex State: progress

Summary

Extended the delta rollout from backend foundations into campaign WebUI high-volume reload paths.
Added a reusable core WebUI delta watermark hook and shared campaign job delta merge helper.
Wired campaign workspace data, campaign report jobs, Review Send inspection/IMAP job loads, and operator queue campaign/summary reloads to delta reads.

Changed Files

govoplan-core/webui/src/utils/deltaHooks.ts
govoplan-core/webui/src/index.ts
govoplan-campaign/webui/src/features/campaigns/utils/jobDeltas.ts
govoplan-campaign/webui/src/features/campaigns/hooks/useCampaignWorkspaceData.ts
govoplan-campaign/webui/src/features/campaigns/CampaignReportPage.tsx
govoplan-campaign/webui/src/features/campaigns/ReviewSendPage.tsx
govoplan-campaign/webui/src/features/operator/OperatorQueuePage.tsx

Verification

govoplan-core/webui: npm run build
govoplan-campaign/webui: npm run test:policy-ui
govoplan-campaign/webui: npm run test:template-preview
govoplan-campaign/webui: npm run test:import-utils

Next / Blocked

Next target: continue the same pattern into other table-heavy module screens after validating these campaign paths in a browser session.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Extended the delta rollout from backend foundations into campaign WebUI high-volume reload paths. - Added a reusable core WebUI delta watermark hook and shared campaign job delta merge helper. - Wired campaign workspace data, campaign report jobs, Review Send inspection/IMAP job loads, and operator queue campaign/summary reloads to delta reads. ### Changed Files - `govoplan-core/webui/src/utils/deltaHooks.ts` - `govoplan-core/webui/src/index.ts` - `govoplan-campaign/webui/src/features/campaigns/utils/jobDeltas.ts` - `govoplan-campaign/webui/src/features/campaigns/hooks/useCampaignWorkspaceData.ts` - `govoplan-campaign/webui/src/features/campaigns/CampaignReportPage.tsx` - `govoplan-campaign/webui/src/features/campaigns/ReviewSendPage.tsx` - `govoplan-campaign/webui/src/features/operator/OperatorQueuePage.tsx` ### Verification - `govoplan-core/webui: npm run build` - `govoplan-campaign/webui: npm run test:policy-ui` - `govoplan-campaign/webui: npm run test:template-preview` - `govoplan-campaign/webui: npm run test:import-utils` ### Next / Blocked - Next target: continue the same pattern into other table-heavy module screens after validating these campaign paths in a browser session. Suggested status label: `status/in-progress`

zemion commented

2026-07-09 13:33:04 +02:00

Codex State: progress

Summary

Extended the conditional/delta foundation into access administration: users, groups, tenant roles, system roles, system accounts, and tenant API keys now expose delta endpoints with watermarks and tombstones.
Access admin WebUI high-volume table panels now merge delta responses with local rows instead of refetching full lists after every write.
For settings-style pages: unchanged resources benefit from ETags already; true one-field updates need section/field-level revision semantics and should be the next core-wide settings retrofit.

Changed Files

govoplan-access/src/govoplan_access/backend/api/v1/routes.py
govoplan-access/src/govoplan_access/backend/api/v1/admin_schemas.py
govoplan-access/webui/src/api/admin.ts
govoplan-access/webui/src/features/admin/utils/deltaRows.ts
govoplan-access/webui/src/features/admin/UsersPanel.tsx
govoplan-access/webui/src/features/admin/GroupsPanel.tsx
govoplan-access/webui/src/features/admin/RolesPanel.tsx
govoplan-access/webui/src/features/admin/SystemRolesPanel.tsx
govoplan-access/webui/src/features/admin/SystemUsersPanel.tsx
govoplan-access/webui/src/features/admin/ApiKeysPanel.tsx
govoplan-core/tests/test_api_smoke.py

Verification

py_compile access routes/schemas and updated smoke test file: pass
python -m unittest tests.test_api_smoke.ApiSmokeTests.test_access_admin_users_delta_tracks_create_and_update tests.test_api_smoke.ApiSmokeTests.test_access_admin_api_key_delta_returns_tombstone_when_revoked_hidden: pass
govoplan-core/webui npm run build: pass

Next / Blocked

Remaining delta candidates: system/tenant settings section deltas, audit/configuration-change streams, and selector-only user/group lookups in secondary admin panels.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Extended the conditional/delta foundation into access administration: users, groups, tenant roles, system roles, system accounts, and tenant API keys now expose delta endpoints with watermarks and tombstones. - Access admin WebUI high-volume table panels now merge delta responses with local rows instead of refetching full lists after every write. - For settings-style pages: unchanged resources benefit from ETags already; true one-field updates need section/field-level revision semantics and should be the next core-wide settings retrofit. ### Changed Files - `govoplan-access/src/govoplan_access/backend/api/v1/routes.py` - `govoplan-access/src/govoplan_access/backend/api/v1/admin_schemas.py` - `govoplan-access/webui/src/api/admin.ts` - `govoplan-access/webui/src/features/admin/utils/deltaRows.ts` - `govoplan-access/webui/src/features/admin/UsersPanel.tsx` - `govoplan-access/webui/src/features/admin/GroupsPanel.tsx` - `govoplan-access/webui/src/features/admin/RolesPanel.tsx` - `govoplan-access/webui/src/features/admin/SystemRolesPanel.tsx` - `govoplan-access/webui/src/features/admin/SystemUsersPanel.tsx` - `govoplan-access/webui/src/features/admin/ApiKeysPanel.tsx` - `govoplan-core/tests/test_api_smoke.py` ### Verification - `py_compile access routes/schemas and updated smoke test file: pass` - `python -m unittest tests.test_api_smoke.ApiSmokeTests.test_access_admin_users_delta_tracks_create_and_update tests.test_api_smoke.ApiSmokeTests.test_access_admin_api_key_delta_returns_tombstone_when_revoked_hidden: pass` - `govoplan-core/webui npm run build: pass` ### Next / Blocked - Remaining delta candidates: system/tenant settings section deltas, audit/configuration-change streams, and selector-only user/group lookups in secondary admin panels. Suggested status label: `status/in-progress`

zemion commented

2026-07-09 15:07:13 +02:00

Codex State: progress

Summary

Completed the next delta pass beyond tables: system settings now expose section-level deltas, tenant settings can refresh tenant-local sections and system language-package changes, configuration changes expose request/history deltas, and audit exposes append-only event deltas.
WebUI panels now consume those deltas: system settings, tenant settings, configuration changes, default audit view, and selector-only user/group target lists for retention, mail profiles, and file connectors.
Fixed the shared sequence watermark edge case where an initial seq:0 baseline was incorrectly considered stale after unrelated collections advanced the global sequence.

Changed Files

govoplan-core/src/govoplan_core/core/change_sequence.py
govoplan-core/src/govoplan_core/core/configuration_control.py
govoplan-core/src/govoplan_core/audit/logging.py
govoplan-core/tests/test_api_smoke.py
govoplan-admin/src/govoplan_admin/backend/api/v1/routes.py
govoplan-admin/src/govoplan_admin/backend/api/v1/schemas.py
govoplan-admin/webui/src/api/admin.ts
govoplan-admin/webui/src/features/admin/SystemSettingsPanel.tsx
govoplan-admin/webui/src/features/admin/ConfigurationChangesPanel.tsx
govoplan-tenancy/src/govoplan_tenancy/backend/api/v1/routes.py
govoplan-tenancy/src/govoplan_tenancy/backend/api/v1/schemas.py
govoplan-audit/src/govoplan_audit/backend/api/v1/routes.py
govoplan-audit/src/govoplan_audit/backend/api/v1/schemas.py
govoplan-access/src/govoplan_access/backend/api/v1/routes.py
govoplan-access/src/govoplan_access/backend/api/v1/admin_schemas.py
govoplan-access/webui/src/api/admin.ts
govoplan-access/webui/src/features/admin/AdminAuditPanel.tsx
govoplan-access/webui/src/features/admin/TenantSettingsPanel.tsx
govoplan-access/webui/src/features/admin/RetentionPoliciesPanel.tsx
govoplan-access/webui/src/features/admin/MailProfilesPanel.tsx
govoplan-access/webui/src/features/admin/FileConnectorsPanel.tsx

Verification

py_compile changed backend modules and tests: pass
python -m unittest tests.test_api_smoke.ApiSmokeTests.test_access_admin_users_delta_tracks_create_and_update tests.test_api_smoke.ApiSmokeTests.test_access_admin_api_key_delta_returns_tombstone_when_revoked_hidden tests.test_api_smoke.ApiSmokeTests.test_settings_deltas_track_sections_and_system_language_dependency tests.test_api_smoke.ApiSmokeTests.test_configuration_changes_delta_tracks_requests tests.test_api_smoke.ApiSmokeTests.test_admin_audit_delta_tracks_new_events: pass
govoplan-core/webui npm run build: pass
git diff --check on touched files only: pass; full worktree still has unrelated pre-existing whitespace noise

Next / Blocked

Remaining follow-up is mostly polish/coverage: decide whether non-default audit table filters should get explicit stream semantics, and later add section deltas to additional module-specific settings pages as they appear.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Completed the next delta pass beyond tables: system settings now expose section-level deltas, tenant settings can refresh tenant-local sections and system language-package changes, configuration changes expose request/history deltas, and audit exposes append-only event deltas. - WebUI panels now consume those deltas: system settings, tenant settings, configuration changes, default audit view, and selector-only user/group target lists for retention, mail profiles, and file connectors. - Fixed the shared sequence watermark edge case where an initial seq:0 baseline was incorrectly considered stale after unrelated collections advanced the global sequence. ### Changed Files - `govoplan-core/src/govoplan_core/core/change_sequence.py` - `govoplan-core/src/govoplan_core/core/configuration_control.py` - `govoplan-core/src/govoplan_core/audit/logging.py` - `govoplan-core/tests/test_api_smoke.py` - `govoplan-admin/src/govoplan_admin/backend/api/v1/routes.py` - `govoplan-admin/src/govoplan_admin/backend/api/v1/schemas.py` - `govoplan-admin/webui/src/api/admin.ts` - `govoplan-admin/webui/src/features/admin/SystemSettingsPanel.tsx` - `govoplan-admin/webui/src/features/admin/ConfigurationChangesPanel.tsx` - `govoplan-tenancy/src/govoplan_tenancy/backend/api/v1/routes.py` - `govoplan-tenancy/src/govoplan_tenancy/backend/api/v1/schemas.py` - `govoplan-audit/src/govoplan_audit/backend/api/v1/routes.py` - `govoplan-audit/src/govoplan_audit/backend/api/v1/schemas.py` - `govoplan-access/src/govoplan_access/backend/api/v1/routes.py` - `govoplan-access/src/govoplan_access/backend/api/v1/admin_schemas.py` - `govoplan-access/webui/src/api/admin.ts` - `govoplan-access/webui/src/features/admin/AdminAuditPanel.tsx` - `govoplan-access/webui/src/features/admin/TenantSettingsPanel.tsx` - `govoplan-access/webui/src/features/admin/RetentionPoliciesPanel.tsx` - `govoplan-access/webui/src/features/admin/MailProfilesPanel.tsx` - `govoplan-access/webui/src/features/admin/FileConnectorsPanel.tsx` ### Verification - `py_compile changed backend modules and tests: pass` - `python -m unittest tests.test_api_smoke.ApiSmokeTests.test_access_admin_users_delta_tracks_create_and_update tests.test_api_smoke.ApiSmokeTests.test_access_admin_api_key_delta_returns_tombstone_when_revoked_hidden tests.test_api_smoke.ApiSmokeTests.test_settings_deltas_track_sections_and_system_language_dependency tests.test_api_smoke.ApiSmokeTests.test_configuration_changes_delta_tracks_requests tests.test_api_smoke.ApiSmokeTests.test_admin_audit_delta_tracks_new_events: pass` - `govoplan-core/webui npm run build: pass` - `git diff --check on touched files only: pass; full worktree still has unrelated pre-existing whitespace noise` ### Next / Blocked - Remaining follow-up is mostly polish/coverage: decide whether non-default audit table filters should get explicit stream semantics, and later add section deltas to additional module-specific settings pages as they appear. Suggested status label: `status/in-progress`

zemion commented

2026-07-09 15:15:53 +02:00

Codex State: progress

Summary

Extended the conditional GET/delta API implementation with the remaining concrete hardening.

Changed Files

Audit deltas now support filtered/sorted first-page views.
Core change-sequence now has an explicit retention-floor/pruning contract plus Alembic retention metadata.
Docs now list current consumers, section-level settings deltas, retention-floor rules, and open retrofit scope.

Verification

unittest delta/change-sequence/access/audit focused suite passed.
webui npm run build passed.
git diff --check and migration py_compile passed for touched files.

Next / Blocked

Adopt section-level deltas in module-specific settings endpoints as APIs stabilize.
Use cursor/keyset pagination before expanding row-merge deltas to arbitrary offset pages.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Extended the conditional GET/delta API implementation with the remaining concrete hardening. ### Changed Files - `Audit deltas now support filtered/sorted first-page views.` - `Core change-sequence now has an explicit retention-floor/pruning contract plus Alembic retention metadata.` - `Docs now list current consumers, section-level settings deltas, retention-floor rules, and open retrofit scope.` ### Verification - `unittest delta/change-sequence/access/audit focused suite passed.` - `webui npm run build passed.` - `git diff --check and migration py_compile passed for touched files.` ### Next / Blocked - Adopt section-level deltas in module-specific settings endpoints as APIs stabilize. - Use cursor/keyset pagination before expanding row-merge deltas to arbitrary offset pages. Suggested status label: `status/in-progress`

zemion commented

2026-07-09 15:42:32 +02:00

Codex State: progress

Summary

Implemented the open module-specific settings delta scope for existing mail and files connector settings surfaces.

Changed Files

Added /api/v1/mail/settings/delta with mail profile row deltas, scoped policy refreshes, sequence write hooks, and WebUI consumption in the mail settings panel.
Added /api/v1/files/connectors/settings/delta with connector profile, credential, connector-space, and policy deltas; credential changes include dependent profiles.
Updated module WebUI API clients and settings panels to consume the new deltas; documented the implemented consumers in docs/MODULE_ARCHITECTURE.md.

Verification

Focused unittest delta suite passed, including mail settings and file connector settings deltas.
Core WebUI npm run build passed.
Mail WebUI test:mail-ui passed using the shared core WebUI TypeScript binary.
Python py_compile and git diff --check passed for touched files.

Next / Blocked

Remaining dependency: arbitrary offset-page row deltas still require cursor/keyset pagination before safe frontend merge behavior can be expanded beyond first-page/safe views.

Suggested status label: status/in-progress

## Codex State: progress ### Summary - Implemented the open module-specific settings delta scope for existing mail and files connector settings surfaces. ### Changed Files - `Added /api/v1/mail/settings/delta with mail profile row deltas, scoped policy refreshes, sequence write hooks, and WebUI consumption in the mail settings panel.` - `Added /api/v1/files/connectors/settings/delta with connector profile, credential, connector-space, and policy deltas; credential changes include dependent profiles.` - `Updated module WebUI API clients and settings panels to consume the new deltas; documented the implemented consumers in docs/MODULE_ARCHITECTURE.md.` ### Verification - `Focused unittest delta suite passed, including mail settings and file connector settings deltas.` - `Core WebUI npm run build passed.` - `Mail WebUI test:mail-ui passed using the shared core WebUI TypeScript binary.` - `Python py_compile and git diff --check passed for touched files.` ### Next / Blocked - Remaining dependency: arbitrary offset-page row deltas still require cursor/keyset pagination before safe frontend merge behavior can be expanded beyond first-page/safe views. Suggested status label: `status/in-progress`

zemion commented

2026-07-09 16:05:30 +02:00

Implemented the remaining cursor/keyset dependency for arbitrary-page delta safety.

What changed:

Added shared core cursor helpers in govoplan_core.core.pagination (encode_keyset_cursor, decode_keyset_cursor, keyset_query_fingerprint).
Retrofitted admin audit list/delta responses with cursor and next_cursor while keeping existing page/offset compatibility.
Audit cursors are fingerprinted by scope, tenant, page size, sort, and filters; mismatched reuse fails with a client error.
The Access WebUI audit panel now remembers page cursors, uses cursor-backed delta reloads for known pages, and falls back to full offset loads only for first visits to arbitrary pages.
Added focused utility and audit smoke coverage, including a stable page-2 cursor test after a newer audit event is inserted.

Verification:

./.venv/bin/python -m unittest tests.test_pagination
./.venv/bin/python -m unittest tests.test_api_smoke.ApiSmokeTests.test_admin_audit_cursor_pagination_is_stable_after_newer_insert
adjacent audit smoke tests for offset pagination and first-page delta
./.venv/bin/python -m compileall src/govoplan_core ../govoplan-audit/src/govoplan_audit ../govoplan-access/src/govoplan_access
npm run build in govoplan-core/webui

Remaining rollout is endpoint-by-endpoint adoption of the same cursor contract, with files lists as the next recommended target and campaign jobs after that.

Implemented the remaining cursor/keyset dependency for arbitrary-page delta safety. What changed: - Added shared core cursor helpers in `govoplan_core.core.pagination` (`encode_keyset_cursor`, `decode_keyset_cursor`, `keyset_query_fingerprint`). - Retrofitted admin audit list/delta responses with `cursor` and `next_cursor` while keeping existing `page`/`offset` compatibility. - Audit cursors are fingerprinted by scope, tenant, page size, sort, and filters; mismatched reuse fails with a client error. - The Access WebUI audit panel now remembers page cursors, uses cursor-backed delta reloads for known pages, and falls back to full offset loads only for first visits to arbitrary pages. - Added focused utility and audit smoke coverage, including a stable page-2 cursor test after a newer audit event is inserted. Verification: - `./.venv/bin/python -m unittest tests.test_pagination` - `./.venv/bin/python -m unittest tests.test_api_smoke.ApiSmokeTests.test_admin_audit_cursor_pagination_is_stable_after_newer_insert` - adjacent audit smoke tests for offset pagination and first-page delta - `./.venv/bin/python -m compileall src/govoplan_core ../govoplan-audit/src/govoplan_audit ../govoplan-access/src/govoplan_access` - `npm run build` in `govoplan-core/webui` Remaining rollout is endpoint-by-endpoint adoption of the same cursor contract, with files lists as the next recommended target and campaign jobs after that.

zemion commented

2026-07-09 16:23:53 +02:00

Rollout scan completed. Direct implementation added for campaign jobs, the other high-value candidates are now tracked as module-specific issues.

Implemented now:

govoplan-campaign: campaign job list/delta now supports cursor/keyset paging with cursor and next_cursor; WebUI report/review pages preserve page cursors; smoke coverage added.

Deferred by domain-specific dependency:

govoplan-files#22: large file-space server windows and cursor-safe deltas.
govoplan-mail#9: provider-aware IMAP mailbox cursors/UID semantics.
govoplan-calendar#7: calendar event-window deltas and tombstones.
govoplan-tenancy#1: tenant administration row deltas.
govoplan-admin#2: governance and module-operation list deltas/cursors.

This closes #222 as the platform/foundation issue. Future endpoint work should use the shared contract from the start.

Rollout scan completed. Direct implementation added for campaign jobs, the other high-value candidates are now tracked as module-specific issues. Implemented now: - `govoplan-campaign`: campaign job list/delta now supports cursor/keyset paging with `cursor` and `next_cursor`; WebUI report/review pages preserve page cursors; smoke coverage added. Deferred by domain-specific dependency: - govoplan-files#22: large file-space server windows and cursor-safe deltas. - govoplan-mail#9: provider-aware IMAP mailbox cursors/UID semantics. - govoplan-calendar#7: calendar event-window deltas and tombstones. - govoplan-tenancy#1: tenant administration row deltas. - govoplan-admin#2: governance and module-operation list deltas/cursors. This closes #222 as the platform/foundation issue. Future endpoint work should use the shared contract from the start.

zemion closed this issue

2026-07-09 16:23:54 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: add-ideas/govoplan-core#222