[Feature] Target-state module update planner #231

New Issue

zemion · 2026-07-10T21:27:35+02:00

zemion commented

2026-07-10 21:27:35 +02:00

Problem

Module package changes are currently planned as install/uninstall items with catalog provenance and preflight checks. That is enough for trusted catalog-sourced installs, but not enough for safe module updates where multiple modules must move together.

The risky case is a coupled update path:

module A needs module B upgraded to satisfy a new interface range
module B needs module A upgraded for its own new interface range
both modules have live data and migrations, so an arbitrary sequential upgrade can leave the system in an unsafe intermediate state

The platform needs a first-class update planner that treats updates as a target-state operation, not as one-module-at-a-time runtime toggles.

Current state

Already available:

module manifests with named provides_interfaces / requires_interfaces
release catalog generation from discovered manifests
catalog validation and operator-visible preflight warnings/blockers
install-plan provenance with source: manual|catalog
catalog metadata snapshots on catalog-sourced plan items
production default requiring trusted signed module package catalogs unless explicitly disabled
documented update-path policy in docs/RELEASE_DEPENDENCIES.md

Required behavior

Implement a core update planner that can plan a desired target module version set before activation.

The planner should:

model explicit update operations, not only install/uninstall
read target versions and package refs from the validated package catalog
compare current discovered manifests with target catalog/manifests
solve required module dependencies and named interface ranges against the target set
report blockers when the target set cannot satisfy all non-optional contracts
allow bridge releases by validating compatibility windows rather than forcing an arbitrary package order
prevent circular “upgrade A first / upgrade B first” traps by validating the final target set before activation
include catalog provenance and signature/trust state in the plan/audit trail
integrate with installer preflight and the admin module-management UI

Live data and migrations

The planner must account for live data risk:

migrations must be ordered by module migration metadata
destructive data/schema changes require explicit retirement or cleanup plans
cross-module migrations must be compatible with old and target provider interfaces until activation finishes
rollback must restore package set and database state together, or the plan must be marked forward-only with a tested recovery procedure
mutually incompatible live-data states require an intermediate bridge release, not a circular update chain

Suggested implementation slices

Add update-plan data model support: action update, current/target versions, target refs, catalog metadata.
Add target-set resolver for catalog entries, dependencies, and named interfaces.
Extend installer preflight to validate update plans against target manifests/contracts.
Add migration/data-safety diagnostics and forward-only markers.
Surface update plans in the admin module-management UI with clear blockers and operator guidance.
Add tests for coupled updates, missing bridge releases, compatible bridge releases, and live-data blockers.

Acceptance criteria

Operators can ask for an update of one module and see required companion updates before activation.
A target set with circular intermediate ordering but compatible final interfaces is allowed as one planned update set.
A target set with unsatisfied interface ranges is blocked before installer execution.
Catalog trust failures block catalog-sourced updates.
Migration/data safety risks are visible in preflight and block when destructive or forward-only requirements are not acknowledged.
Admin UI shows the target update set, provenance, blockers, warnings, and required operator steps without requiring JSON editing.

## Problem Module package changes are currently planned as install/uninstall items with catalog provenance and preflight checks. That is enough for trusted catalog-sourced installs, but not enough for safe module updates where multiple modules must move together. The risky case is a coupled update path: - module A needs module B upgraded to satisfy a new interface range - module B needs module A upgraded for its own new interface range - both modules have live data and migrations, so an arbitrary sequential upgrade can leave the system in an unsafe intermediate state The platform needs a first-class update planner that treats updates as a target-state operation, not as one-module-at-a-time runtime toggles. ## Current state Already available: - module manifests with named `provides_interfaces` / `requires_interfaces` - release catalog generation from discovered manifests - catalog validation and operator-visible preflight warnings/blockers - install-plan provenance with `source: manual|catalog` - catalog metadata snapshots on catalog-sourced plan items - production default requiring trusted signed module package catalogs unless explicitly disabled - documented update-path policy in `docs/RELEASE_DEPENDENCIES.md` ## Required behavior Implement a core update planner that can plan a desired target module version set before activation. The planner should: - model explicit update operations, not only install/uninstall - read target versions and package refs from the validated package catalog - compare current discovered manifests with target catalog/manifests - solve required module dependencies and named interface ranges against the target set - report blockers when the target set cannot satisfy all non-optional contracts - allow bridge releases by validating compatibility windows rather than forcing an arbitrary package order - prevent circular “upgrade A first / upgrade B first” traps by validating the final target set before activation - include catalog provenance and signature/trust state in the plan/audit trail - integrate with installer preflight and the admin module-management UI ## Live data and migrations The planner must account for live data risk: - migrations must be ordered by module migration metadata - destructive data/schema changes require explicit retirement or cleanup plans - cross-module migrations must be compatible with old and target provider interfaces until activation finishes - rollback must restore package set and database state together, or the plan must be marked forward-only with a tested recovery procedure - mutually incompatible live-data states require an intermediate bridge release, not a circular update chain ## Suggested implementation slices 1. Add update-plan data model support: action `update`, current/target versions, target refs, catalog metadata. 2. Add target-set resolver for catalog entries, dependencies, and named interfaces. 3. Extend installer preflight to validate update plans against target manifests/contracts. 4. Add migration/data-safety diagnostics and forward-only markers. 5. Surface update plans in the admin module-management UI with clear blockers and operator guidance. 6. Add tests for coupled updates, missing bridge releases, compatible bridge releases, and live-data blockers. ## Acceptance criteria - Operators can ask for an update of one module and see required companion updates before activation. - A target set with circular intermediate ordering but compatible final interfaces is allowed as one planned update set. - A target set with unsatisfied interface ranges is blocked before installer execution. - Catalog trust failures block catalog-sourced updates. - Migration/data safety risks are visible in preflight and block when destructive or forward-only requirements are not acknowledged. - Admin UI shows the target update set, provenance, blockers, warnings, and required operator steps without requiring JSON editing.

zemion commented

2026-07-10 22:24:55 +02:00

First slice implemented locally: install plans now support explicit update rows; catalog planning records update when a module is already installed; package catalog entries preserve dependency metadata; preflight resolves catalog-sourced installs/updates as a target state and blocks activation with companion_update_required when a planned update needs another catalog module update or would break an installed consumer. Docs updated in RELEASE_DEPENDENCIES.md and MODULE_ARCHITECTURE.md. Verification: python -m unittest tests.test_module_system passes.

zemion commented

2026-07-10 22:54:09 +02:00

Next slice implemented locally: release catalog entries can now declare migration_safety (automatic/requires_review/forward_only/destructive) plus migration_notes; generated catalogs mark migration-owning modules as requires_review; install-plan rows can carry data_safety_acknowledged; installer preflight blocks unacknowledged forward-only/destructive catalog updates and requires a cleanup/retirement note for destructive entries. Admin schemas and UI expose the safety metadata and acknowledgement toggle. Verification: python -m unittest tests.test_module_system passes (89 tests).

zemion commented

2026-07-10 23:22:40 +02:00

Next slice implemented locally: installer preflight now returns a structured target_plan summary for planned package changes, including action, source, current version, catalog target version, package refs, migration_safety, migration_notes, and data_safety_acknowledged. Admin API schemas and the module-management UI now surface this target plan above the raw preflight issues so operators can review the intended current->target state without JSON editing. Docs updated in RELEASE_DEPENDENCIES.md and MODULE_ARCHITECTURE.md. Verification: python -m unittest tests.test_module_system passes (90 tests); git diff --check passes in core and admin.

zemion commented

2026-07-10 23:39:32 +02:00

Merged related scope from add-ideas/govoplan-core#35: add-ideas/govoplan-core#35

The older issue asks for WebUI-driven update/rollback with backup and rollback guardrails. Those requirements are part of the target-state update planner and its installer/preflight/rollback guardrail work, so #231 remains canonical.

Merged related scope from `add-ideas/govoplan-core#35`: https://git.add-ideas.de/add-ideas/govoplan-core/issues/35 The older issue asks for WebUI-driven update/rollback with backup and rollback guardrails. Those requirements are part of the target-state update planner and its installer/preflight/rollback guardrail work, so #231 remains canonical.

zemion referenced this issue

2026-07-10 23:39:36 +02:00

[Task] WebUI-driven update/rollback with backup and rollback guardrails #35

zemion commented

2026-07-10 23:40:46 +02:00

Implemented the next update-planner slice.

Done now:

catalog planning can auto-add resolvable companion rows for dependency/interface provider updates and installed consumer updates
catalog preflight blocks downgrade and same-version update plans by default; reviewed catalogs can opt in with allow_downgrade / allow_same_version
catalog preflight enforces current-version update windows and surfaces bridge-release metadata
forward-only/destructive catalog updates now require recovery_tested=true plus recovery notes before acknowledgement is enough
target_plan and package-catalog API/UI now surface update windows, bridge metadata, and recovery metadata
release docs and the example package catalog document the new fields and operator behavior

Validation:

.venv/bin/python -m unittest tests.test_module_system: 98 tests OK
backend py_compile for edited core/admin files OK
npm run build in govoplan-core/webui OK
git diff --check OK in core and admin
docs/module-package-catalog.example.json parses as JSON

Still not part of this slice:

the full live-data migration graph executor / per-module migration step ordering beyond the current target-state preflight and catalog safety gates.

Implemented the next update-planner slice. Done now: - catalog planning can auto-add resolvable companion rows for dependency/interface provider updates and installed consumer updates - catalog preflight blocks downgrade and same-version update plans by default; reviewed catalogs can opt in with allow_downgrade / allow_same_version - catalog preflight enforces current-version update windows and surfaces bridge-release metadata - forward-only/destructive catalog updates now require recovery_tested=true plus recovery notes before acknowledgement is enough - target_plan and package-catalog API/UI now surface update windows, bridge metadata, and recovery metadata - release docs and the example package catalog document the new fields and operator behavior Validation: - .venv/bin/python -m unittest tests.test_module_system: 98 tests OK - backend py_compile for edited core/admin files OK - npm run build in govoplan-core/webui OK - git diff --check OK in core and admin - docs/module-package-catalog.example.json parses as JSON Still not part of this slice: - the full live-data migration graph executor / per-module migration step ordering beyond the current target-state preflight and catalog safety gates.

zemion commented

2026-07-11 00:36:12 +02:00

Implemented the next update-planner slice: module-aware migration execution planning.

Done now:

added manifest/catalog migration ordering metadata: migration_after and migration_before
release catalog normalization and generator preserve those migration ordering fields
preflight now builds a structured migration_plan with target enabled modules and ordered migration steps
migration graph order is derived from explicit migration metadata, module dependencies/optional dependencies, and named interface provider/consumer relationships
preflight blocks migration ordering cycles before an installer request can be queued
installer-triggered migrations now call init_db with the target enabled module set, so newly installed module migration directories are discovered before activation
init_db/migrate_database can receive ordered module heads; the migrator upgrades those module heads first, then finishes with Alembic heads
admin API/WebUI exposes the migration plan and catalog ordering metadata
docs and example catalog updated

Validation:

python -m unittest tests.test_module_system: 101 tests OK
python -m unittest tests.test_database_migrations: 8 tests OK
npm run build in govoplan-core/webui OK
py_compile for edited core/admin Python files OK
git diff --check OK in core and admin
docs/module-package-catalog.example.json parses as JSON

Remaining boundary:

Alembic remains the authoritative revision executor. This slice adds module-aware ordering and target-module discovery around it; it does not yet introduce module-owned pre/post live-data hooks outside Alembic revisions.

Implemented the next update-planner slice: module-aware migration execution planning. Done now: - added manifest/catalog migration ordering metadata: migration_after and migration_before - release catalog normalization and generator preserve those migration ordering fields - preflight now builds a structured migration_plan with target enabled modules and ordered migration steps - migration graph order is derived from explicit migration metadata, module dependencies/optional dependencies, and named interface provider/consumer relationships - preflight blocks migration ordering cycles before an installer request can be queued - installer-triggered migrations now call init_db with the target enabled module set, so newly installed module migration directories are discovered before activation - init_db/migrate_database can receive ordered module heads; the migrator upgrades those module heads first, then finishes with Alembic heads - admin API/WebUI exposes the migration plan and catalog ordering metadata - docs and example catalog updated Validation: - python -m unittest tests.test_module_system: 101 tests OK - python -m unittest tests.test_database_migrations: 8 tests OK - npm run build in govoplan-core/webui OK - py_compile for edited core/admin Python files OK - git diff --check OK in core and admin - docs/module-package-catalog.example.json parses as JSON Remaining boundary: - Alembic remains the authoritative revision executor. This slice adds module-aware ordering and target-module discovery around it; it does not yet introduce module-owned pre/post live-data hooks outside Alembic revisions.

zemion commented

2026-07-11 01:13:02 +02:00

Implemented the remaining live-data migration task slice for the target-state update planner.

What is now covered:

MigrationSpec supports constrained migration_tasks with explicit phases, task version, safety, idempotency, optional timeout, and executor metadata.
Release catalog validation and scripts/generate-release-catalog.py preserve task metadata so operators see warnings before activation.
Installer preflight exposes migration_plan.tasks and blocks unsafe activation:
- non-idempotent tasks block
- forward-only/destructive tasks require operator acknowledgement
- installed manifest tasks without executors block
- catalog-only executors are marked pending until package install verification
govoplan_core.commands.init_db runs pre-migration tasks before Alembic and post-migration tasks after Alembic, writing task records back to installer run records.
Admin module-management API/UI surfaces migration tasks alongside ordered migration steps.
Documentation and catalog example were updated.

Validation run:

python -m unittest tests.test_module_system OK
python -m unittest tests.test_database_migrations OK
npm run build in govoplan-core/webui OK
Python compile checks OK
git diff --check OK for core/admin

I am closing this because the target-state planner, companion update resolution, catalog trust/contract validation, migration ordering, live-data task diagnostics/execution, and admin preflight surface are now implemented. Any concrete module-specific migration task can be tracked separately.

Implemented the remaining live-data migration task slice for the target-state update planner. What is now covered: - `MigrationSpec` supports constrained `migration_tasks` with explicit phases, task version, safety, idempotency, optional timeout, and executor metadata. - Release catalog validation and `scripts/generate-release-catalog.py` preserve task metadata so operators see warnings before activation. - Installer preflight exposes `migration_plan.tasks` and blocks unsafe activation: - non-idempotent tasks block - forward-only/destructive tasks require operator acknowledgement - installed manifest tasks without executors block - catalog-only executors are marked pending until package install verification - `govoplan_core.commands.init_db` runs pre-migration tasks before Alembic and post-migration tasks after Alembic, writing task records back to installer run records. - Admin module-management API/UI surfaces migration tasks alongside ordered migration steps. - Documentation and catalog example were updated. Validation run: - `python -m unittest tests.test_module_system` OK - `python -m unittest tests.test_database_migrations` OK - `npm run build` in `govoplan-core/webui` OK - Python compile checks OK - `git diff --check` OK for core/admin I am closing this because the target-state planner, companion update resolution, catalog trust/contract validation, migration ordering, live-data task diagnostics/execution, and admin preflight surface are now implemented. Any concrete module-specific migration task can be tracked separately.

zemion closed this issue

2026-07-11 01:13:02 +02:00

zemion commented

2026-07-11 01:15:16 +02:00

Follow-up refinement after the closing note: failed module migration task execution now raises a task-execution error carrying partial records, and init_db writes the migration task record file from a finally block. That keeps installer/audit evidence available even when a pre/post migration task blocks or fails.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GovOPlaN/govoplan-core#231