feat(release): persist resumable run state

docs/RELEASE_CONSOLE.md

@@ -18,6 +18,7 @@ candidate/publish actions:
 - configure target versions per release unit in the web UI
 - select the repositories that should advance through checkboxes
 - generate dry-run selective release plans for independently versioned packages
+- freeze a selective plan as a durable, resumable local release run
 - create annotated source release tags for selected clean, version-aligned
   repositories and publish each branch/tag pair atomically
 - generate signed catalog candidates for the selected release units
@@ -49,6 +50,70 @@ tag. `source_preflight_ready` means that the plan-visible source gates pass; the
 non-mutating `Preview Tag + Publish` remains mandatory for remote, manifest, and
 immutable-tag checks.
 
+## Durable release runs
+
+The **Durable Run State** card turns the current repository/version selection
+into a versioned local run record. The server rebuilds the selective plan and
+requires the plan to resolve exactly the requested repositories and target
+versions; the browser cannot submit or replace the plan snapshot. The input and
+plan are then immutable and covered by a canonical SHA-256 integrity digest.
+The complete record also has a checksum so a valid-looking manual edit to its
+mutable state fails closed. File permissions remain the authority boundary;
+these digests detect accidental or manual corruption, not an attacker who can
+replace the private record and recompute its checksums. Changing a target,
+channel, or gate input requires a new run.
+
+Run records live below `runtime/release-runs/`. They survive console restarts,
+but remain local operator state rather than a signed release artifact or the
+system audit log. The directory is restricted to mode `0700` and records to
+`0600`. Writes use a cross-process lock, a same-directory temporary file,
+`fsync`, atomic replacement, and directory `fsync`. Symbolic-link storage paths,
+overly broad record modes, malformed schemas, unknown fields, invalid state
+combinations, oversized files, and digest mismatches fail closed. A bad record
+is neither rewritten nor automatically quarantined.
+
+The default store belongs to the selected workspace: the console uses that
+workspace's meta checkout when it is available, or a stable workspace-digest
+namespace below this checkout's runtime directory otherwise. The same digest is
+part of the immutable input snapshot. An alternate workspace therefore cannot
+silently list or resume another workspace's runs. `run_state_root` remains an
+explicit embedding/test override.
+
+Each frozen plan step has an explicit `pending`, `running`, `succeeded`,
+`failed`, or `interrupted` state. Plan order remains a prerequisite: a later
+step is unavailable until earlier steps have succeeded. Exact attempt and
+resume/retry request identifiers are fingerprinted so repeated state commands
+are idempotent. The record keeps at most 256 server-generated state events and
+128 request fingerprints. Events contain only an enum event type, timestamp,
+step identifier, and bounded result code—never commands, process output,
+confirmation text, credentials, bearer tokens, or signing material.
+
+An explicit resume after a process restart converts every persisted `running`
+step to `interrupted`; it never guesses whether an external effect happened.
+An interrupted read-only step can be prepared for retry. An interrupted
+mutating step remains unavailable until the operator has reconciled local and
+remote state and explicitly records that reconciliation when preparing the
+retry. A known failed attempt can likewise be prepared for retry. The UI keeps
+the retry control visible and explains why it is disabled when retry is unsafe.
+
+The run API is covered by the same local console token middleware as every
+other `/api/` route:
+
+- `POST /api/release-runs` rebuilds and freezes a selective plan.
+- `GET /api/release-runs` lists bounded summaries; unreadable entries are
+  reported as unavailable instead of being silently omitted.
+- `GET /api/release-runs/{run_id}` reads and verifies one exact record.
+- `POST /api/release-runs/{run_id}/resume` records explicit recovery.
+- `POST /api/release-runs/{run_id}/steps/{step_id}/retry` prepares a failed or
+  safely reconciled interrupted step for another attempt.
+
+This foundation tracks state only. It deliberately does not run a plan step or
+infer success from a button click. Existing preview and executor controls stay
+separate and continue to require `COMMIT`, `TAG`, `PUBLISH`, `APPLY`, or `PUSH`
+at their existing narrow boundaries. Wiring confirmed executors to claim and
+finish run steps, plus package/install verification, remains follow-up work;
+until then the console must not present a run as execution evidence.
+
 Dashboard collection is also fail closed. Unreadable Core version metadata or a
 malformed module contract is returned as a bounded `collection_errors` entry
 with a remediation, marks the dashboard blocked, and becomes a structured

tests/test_release_run_state.py

@@ -0,0 +1,520 @@
+from __future__ import annotations
+
+from concurrent.futures import ThreadPoolExecutor
+import json
+import os
+from pathlib import Path
+import sys
+import tempfile
+import threading
+import unittest
+from unittest.mock import patch
+
+from fastapi.testclient import TestClient
+
+
+META_ROOT = Path(__file__).resolve().parents[1]
+RELEASE_ROOT = META_ROOT / "tools" / "release"
+if str(RELEASE_ROOT) not in sys.path:
+    sys.path.insert(0, str(RELEASE_ROOT))
+
+from govoplan_release.release_run import (  # noqa: E402
+    MAX_EVENTS,
+    ReleaseRunConflict,
+    ReleaseRunCorrupt,
+    ReleaseRunStore,
+)
+from server.app import (  # noqa: E402
+    create_app,
+    default_release_run_root,
+    release_workspace_fingerprint,
+)
+
+
+class ReleaseRunStoreTests(unittest.TestCase):
+    def setUp(self) -> None:
+        self.temporary = tempfile.TemporaryDirectory()
+        self.root = Path(self.temporary.name) / "release-runs"
+        self.store = ReleaseRunStore(self.root)
+
+    def tearDown(self) -> None:
+        self.temporary.cleanup()
+
+    def test_create_persists_private_immutable_plan_and_ordered_steps(self) -> None:
+        run = self.store.create(
+            input_snapshot=run_input(), plan_snapshot=release_plan()
+        )
+
+        reopened = ReleaseRunStore(self.root).get(run["run_id"])
+        record_path = self.root / f"{run['run_id']}.json"
+
+        self.assertEqual("govoplan.release-run", reopened["schema"])
+        self.assertEqual(1, reopened["schema_version"])
+        self.assertEqual(
+            {"govoplan-core": "1.2.3", "govoplan-files": "1.2.4"},
+            reopened["immutable"]["input"]["repo_versions"],
+        )
+        self.assertEqual("planned", reopened["state"]["status"])
+        self.assertTrue(reopened["state"]["steps"][0]["available"])
+        self.assertFalse(reopened["state"]["steps"][1]["available"])
+        self.assertEqual("core:preflight", reopened["recommended_next"]["step_id"])
+        self.assertEqual(0o700, os.stat(self.root).st_mode & 0o777)
+        self.assertEqual(0o600, os.stat(record_path).st_mode & 0o777)
+        self.assertNotIn("processed_requests", reopened["state"])
+        self.assertNotIn("attempt_fingerprint", reopened["state"]["steps"][0])
+
+    def test_attempt_transitions_are_idempotent_and_strictly_ordered(self) -> None:
+        run_id = self.store.create(
+            input_snapshot=run_input(), plan_snapshot=release_plan()
+        )["run_id"]
+
+        with self.assertRaisesRegex(ReleaseRunConflict, "Complete core:preflight"):
+            self.store.start_step(run_id, "core:tag", attempt_id="attempt-tag-0001")
+
+        started = self.store.start_step(
+            run_id, "core:preflight", attempt_id="attempt-preflight-0001"
+        )
+        duplicate_start = self.store.start_step(
+            run_id, "core:preflight", attempt_id="attempt-preflight-0001"
+        )
+        self.assertEqual(1, started["state"]["steps"][0]["attempt_count"])
+        self.assertEqual(
+            len(started["state"]["events"]),
+            len(duplicate_start["state"]["events"]),
+        )
+
+        finished = self.store.finish_step(
+            run_id,
+            "core:preflight",
+            attempt_id="attempt-preflight-0001",
+            succeeded=True,
+            result_code="preflight_valid",
+        )
+        duplicate_finish = self.store.finish_step(
+            run_id,
+            "core:preflight",
+            attempt_id="attempt-preflight-0001",
+            succeeded=True,
+            result_code="preflight_valid",
+        )
+        self.assertEqual("succeeded", finished["state"]["steps"][0]["state"])
+        self.assertEqual(
+            len(finished["state"]["events"]),
+            len(duplicate_finish["state"]["events"]),
+        )
+        self.assertTrue(finished["state"]["steps"][1]["available"])
+
+    def test_reopen_and_resume_preserve_ambiguous_mutating_effect(self) -> None:
+        run_id = self.store.create(
+            input_snapshot=run_input(), plan_snapshot=mutating_first_plan()
+        )["run_id"]
+        self.store.start_step(run_id, "core:tag", attempt_id="attempt-tag-0001")
+
+        reopened_store = ReleaseRunStore(self.root)
+        before_resume = reopened_store.get(run_id)
+        self.assertEqual("running", before_resume["state"]["steps"][0]["state"])
+
+        resumed = reopened_store.resume(run_id, request_id="resume-request-0001")
+        duplicate = reopened_store.resume(run_id, request_id="resume-request-0001")
+        self.assertEqual("interrupted", resumed["state"]["steps"][0]["state"])
+        self.assertEqual("reconcile_step", resumed["recommended_next"]["id"])
+        self.assertFalse(resumed["recommended_next"]["available"])
+        self.assertEqual(
+            len(resumed["state"]["events"]), len(duplicate["state"]["events"])
+        )
+
+        with self.assertRaisesRegex(ReleaseRunConflict, "must be reconciled"):
+            reopened_store.retry_step(
+                run_id,
+                "core:tag",
+                request_id="retry-request-0001",
+            )
+        retried = reopened_store.retry_step(
+            run_id,
+            "core:tag",
+            request_id="retry-request-0002",
+            reconciled=True,
+        )
+        duplicate_retry = reopened_store.retry_step(
+            run_id,
+            "core:tag",
+            request_id="retry-request-0002",
+            reconciled=True,
+        )
+        self.assertEqual("pending", retried["state"]["steps"][0]["state"])
+        self.assertEqual(
+            len(retried["state"]["events"]),
+            len(duplicate_retry["state"]["events"]),
+        )
+
+    def test_tampered_record_fails_closed_and_is_not_rewritten(self) -> None:
+        run_id = self.store.create(
+            input_snapshot=run_input(), plan_snapshot=release_plan()
+        )["run_id"]
+        path = self.root / f"{run_id}.json"
+        payload = json.loads(path.read_text(encoding="utf-8"))
+        payload["immutable"]["input"]["channel"] = "testing"
+        path.write_text(json.dumps(payload), encoding="utf-8")
+        tampered = path.read_bytes()
+
+        with self.assertRaises(ReleaseRunCorrupt):
+            self.store.get(run_id)
+
+        self.assertEqual(tampered, path.read_bytes())
+        self.assertEqual("unavailable", self.store.list()[0]["status"])
+
+    def test_broad_record_mode_and_symlinked_root_fail_closed(self) -> None:
+        run_id = self.store.create(
+            input_snapshot=run_input(), plan_snapshot=release_plan()
+        )["run_id"]
+        path = self.root / f"{run_id}.json"
+        path.chmod(0o640)
+        with self.assertRaisesRegex(ReleaseRunCorrupt, "broader than 0600"):
+            self.store.get(run_id)
+        self.assertEqual(0o640, os.stat(path).st_mode & 0o777)
+
+        target = Path(self.temporary.name) / "target"
+        target.mkdir()
+        linked = Path(self.temporary.name) / "linked"
+        linked.symlink_to(target, target_is_directory=True)
+        with self.assertRaisesRegex(ReleaseRunCorrupt, "symbolic-link"):
+            ReleaseRunStore(linked).list()
+
+    def test_state_specific_fields_and_boolean_counts_fail_closed(self) -> None:
+        run_id = self.store.create(
+            input_snapshot=run_input(), plan_snapshot=release_plan()
+        )["run_id"]
+        path = self.root / f"{run_id}.json"
+        payload = json.loads(path.read_text(encoding="utf-8"))
+        payload["state"]["steps"][0]["attempt_count"] = True
+        payload["state"]["steps"][0]["started_at"] = "2026-07-22T00:00:00Z"
+        path.write_text(json.dumps(payload), encoding="utf-8")
+        path.chmod(0o600)
+
+        with self.assertRaises(ReleaseRunCorrupt):
+            self.store.get(run_id)
+
+    def test_valid_looking_mutable_state_tampering_fails_record_checksum(self) -> None:
+        run_id = self.store.create(
+            input_snapshot=run_input(), plan_snapshot=mutating_first_plan()
+        )["run_id"]
+        self.store.start_step(run_id, "core:tag", attempt_id="checksum-attempt-0001")
+        self.store.finish_step(
+            run_id,
+            "core:tag",
+            attempt_id="checksum-attempt-0001",
+            succeeded=True,
+            result_code="tag_created",
+        )
+        path = self.root / f"{run_id}.json"
+        payload = json.loads(path.read_text(encoding="utf-8"))
+        payload["state"]["steps"][0]["result_code"] = "tag_published"
+        path.write_text(json.dumps(payload), encoding="utf-8")
+        path.chmod(0o600)
+        tampered = path.read_bytes()
+
+        with self.assertRaises(ReleaseRunCorrupt):
+            self.store.get(run_id)
+
+        self.assertEqual(tampered, path.read_bytes())
+
+    def test_concurrent_claims_have_one_winner_and_no_lost_update(self) -> None:
+        run_id = self.store.create(
+            input_snapshot=run_input(), plan_snapshot=release_plan()
+        )["run_id"]
+        barrier = threading.Barrier(2)
+
+        def claim(attempt_id: str) -> str:
+            contender = ReleaseRunStore(self.root)
+            barrier.wait(timeout=5)
+            try:
+                contender.start_step(run_id, "core:preflight", attempt_id=attempt_id)
+            except ReleaseRunConflict:
+                return "conflict"
+            return "started"
+
+        with ThreadPoolExecutor(max_workers=2) as executor:
+            results = list(
+                executor.map(
+                    claim, ("concurrent-attempt-0001", "concurrent-attempt-0002")
+                )
+            )
+
+        reopened = self.store.get(run_id)
+        self.assertEqual(["conflict", "started"], sorted(results))
+        self.assertEqual(1, reopened["state"]["steps"][0]["attempt_count"])
+        self.assertEqual(
+            1,
+            sum(
+                event["type"] == "step_started" for event in reopened["state"]["events"]
+            ),
+        )
+
+    def test_event_and_request_history_are_bounded_and_code_only(self) -> None:
+        run_id = self.store.create(
+            input_snapshot=run_input(), plan_snapshot=release_plan()
+        )["run_id"]
+        for index in range(90):
+            attempt_id = f"bounded-attempt-{index:04d}"
+            self.store.start_step(run_id, "core:preflight", attempt_id=attempt_id)
+            self.store.finish_step(
+                run_id,
+                "core:preflight",
+                attempt_id=attempt_id,
+                succeeded=False,
+                result_code="preflight_failed",
+            )
+            self.store.retry_step(
+                run_id,
+                "core:preflight",
+                request_id=f"bounded-retry-{index:04d}",
+            )
+
+        view = self.store.get(run_id)
+        raw = json.loads((self.root / f"{run_id}.json").read_text(encoding="utf-8"))
+        self.assertEqual(MAX_EVENTS, len(view["state"]["events"]))
+        self.assertLessEqual(len(raw["state"]["processed_requests"]), 128)
+        self.assertNotIn("bounded-retry-0089", json.dumps(raw))
+        for event in view["state"]["events"]:
+            self.assertLessEqual(
+                set(event), {"sequence", "at", "type", "step_id", "result_code"}
+            )
+
+
+class ReleaseRunApiTests(unittest.TestCase):
+    def test_token_guarded_create_list_read_resume_and_retry(self) -> None:
+        with tempfile.TemporaryDirectory() as temp_dir:
+            root = Path(temp_dir) / "runs"
+            app = create_app(
+                workspace_root=Path(temp_dir),
+                run_state_root=root,
+                token="release-token",
+            )
+            headers = {"X-Release-Console-Token": "release-token"}
+            with (
+                patch("server.app.build_dashboard", return_value=object()),
+                patch(
+                    "server.app.build_selective_release_plan",
+                    return_value=release_plan(),
+                ),
+                TestClient(app) as client,
+            ):
+                unauthorized = client.get("/api/release-runs")
+                created_response = client.post(
+                    "/api/release-runs",
+                    headers=headers,
+                    json=run_input(),
+                )
+                self.assertEqual(200, created_response.status_code)
+                created = created_response.json()
+                run_id = created["run_id"]
+                listed = client.get("/api/release-runs", headers=headers).json()
+                loaded = client.get(
+                    f"/api/release-runs/{run_id}", headers=headers
+                ).json()
+
+                app.state.release_runs.start_step(
+                    run_id,
+                    "core:preflight",
+                    attempt_id="api-attempt-0001",
+                )
+                resumed = client.post(
+                    f"/api/release-runs/{run_id}/resume",
+                    headers=headers,
+                    json={"request_id": "api-resume-0001"},
+                )
+                retried = client.post(
+                    f"/api/release-runs/{run_id}/steps/core:preflight/retry",
+                    headers=headers,
+                    json={"request_id": "api-retry-0001"},
+                )
+
+            self.assertEqual(401, unauthorized.status_code)
+            self.assertEqual(run_id, listed["runs"][0]["run_id"])
+            self.assertEqual(run_id, loaded["run_id"])
+            self.assertEqual(
+                "interrupted", resumed.json()["state"]["steps"][0]["state"]
+            )
+            self.assertEqual(200, retried.status_code)
+            self.assertEqual("pending", retried.json()["state"]["steps"][0]["state"])
+
+    def test_create_rejects_unknown_or_unresolved_repository(self) -> None:
+        with tempfile.TemporaryDirectory() as temp_dir:
+            root = Path(temp_dir) / "runs"
+            app = create_app(
+                workspace_root=Path(temp_dir), run_state_root=root, token="token"
+            )
+            with (
+                patch("server.app.build_dashboard", return_value=object()),
+                patch(
+                    "server.app.build_selective_release_plan",
+                    return_value=release_plan(),
+                ),
+                TestClient(app) as client,
+            ):
+                response = client.post(
+                    "/api/release-runs",
+                    headers={"X-Release-Console-Token": "token"},
+                    json={
+                        "channel": "stable",
+                        "repo_versions": {"govoplan-unknown": "1.0.0"},
+                        "public_catalog": False,
+                    },
+                )
+
+            self.assertEqual(409, response.status_code)
+            self.assertIn("did not resolve exactly", response.json()["detail"])
+            self.assertEqual([], list(root.glob("*.json")))
+
+    def test_api_rejects_tampered_record_without_rewriting_it(self) -> None:
+        with tempfile.TemporaryDirectory() as temp_dir:
+            root = Path(temp_dir) / "runs"
+            store = ReleaseRunStore(root)
+            run_id = store.create(
+                input_snapshot=run_input(), plan_snapshot=release_plan()
+            )["run_id"]
+            path = root / f"{run_id}.json"
+            path.write_text("{}", encoding="utf-8")
+            malformed = path.read_bytes()
+            app = create_app(
+                workspace_root=Path(temp_dir), run_state_root=root, token="token"
+            )
+            with TestClient(app) as client:
+                response = client.get(
+                    f"/api/release-runs/{run_id}",
+                    headers={"X-Release-Console-Token": "token"},
+                )
+
+            self.assertEqual(409, response.status_code)
+            self.assertEqual(malformed, path.read_bytes())
+
+    def test_default_storage_and_immutable_input_are_workspace_bound(self) -> None:
+        with tempfile.TemporaryDirectory() as temp_dir:
+            parent = Path(temp_dir)
+            first = parent / "first"
+            second = parent / "second"
+            first.mkdir()
+            second.mkdir()
+            first_root = default_release_run_root(first)
+            second_root = default_release_run_root(second)
+            self.assertNotEqual(first_root, second_root)
+            self.assertIn(release_workspace_fingerprint(first)[:16], first_root.name)
+
+            local_workspace = parent / "local"
+            local_meta = local_workspace / "govoplan"
+            local_meta.mkdir(parents=True)
+            (local_meta / "repositories.json").write_text("{}", encoding="utf-8")
+            self.assertEqual(
+                local_meta / "runtime" / "release-runs",
+                default_release_run_root(local_workspace),
+            )
+
+            explicit_root = parent / "explicit"
+            app = create_app(
+                workspace_root=first,
+                run_state_root=explicit_root,
+                token="token",
+            )
+            with (
+                patch("server.app.build_dashboard", return_value=object()),
+                patch(
+                    "server.app.build_selective_release_plan",
+                    return_value=release_plan(),
+                ),
+                TestClient(app) as client,
+            ):
+                response = client.post(
+                    "/api/release-runs",
+                    headers={"X-Release-Console-Token": "token"},
+                    json=run_input(),
+                )
+            self.assertEqual(200, response.status_code)
+            self.assertEqual(
+                release_workspace_fingerprint(first),
+                response.json()["immutable"]["input"]["workspace_fingerprint"],
+            )
+            self.assertEqual(explicit_root.resolve(), app.state.release_runs.root)
+
+    def test_ui_and_runbook_state_tracking_boundary_are_explicit(self) -> None:
+        webui = (RELEASE_ROOT / "webui" / "index.html").read_text(encoding="utf-8")
+        runbook = (META_ROOT / "docs" / "RELEASE_CONSOLE.md").read_text(
+            encoding="utf-8"
+        )
+
+        self.assertIn("Durable Run State", webui)
+        self.assertIn("State tracking only.", webui)
+        self.assertIn("Create Run from Selection", webui)
+        self.assertIn("Prepare Retry", webui)
+        self.assertIn("step.retry_available", webui)
+        self.assertIn("This foundation tracks state only.", runbook)
+        self.assertIn("mutating step remains unavailable", runbook)
+        self.assertIn("same-directory temporary file", runbook)
+
+
+def run_input() -> dict[str, object]:
+    return {
+        "channel": "stable",
+        "repo_versions": {
+            "govoplan-files": "1.2.4",
+            "govoplan-core": "1.2.3",
+        },
+        "online": False,
+        "remote_tags": False,
+        "public_catalog": False,
+        "include_migrations": True,
+        "workspace_fingerprint": "a" * 64,
+    }
+
+
+def release_plan() -> dict[str, object]:
+    return {
+        "generated_at": "2026-07-22T00:00:00Z",
+        "target_channel": "stable",
+        "status": "attention",
+        "units": [
+            {"repo": "govoplan-core", "target_version": "1.2.3"},
+            {"repo": "govoplan-files", "target_version": "1.2.4"},
+        ],
+        "compatibility": [],
+        "gate_findings": [],
+        "recommended_action": {
+            "id": "preview_source_release",
+            "title": "Preview source release tags",
+            "detail": "Plan-visible gates pass.",
+            "remediation": "Preview before mutation.",
+        },
+        "source_preflight_ready": True,
+        "dry_run_steps": [
+            {
+                "id": "core:preflight",
+                "title": "Inspect Core",
+                "detail": "Run a read-only preflight.",
+                "command": "git status --short --branch",
+                "cwd": "/workspace/govoplan-core",
+                "mutating": False,
+                "repo": "govoplan-core",
+                "status": "planned",
+            },
+            {
+                "id": "core:tag",
+                "title": "Create Core tag",
+                "detail": "Create an annotated tag after confirmation.",
+                "command": "git tag -a v1.2.3",
+                "cwd": "/workspace/govoplan-core",
+                "mutating": True,
+                "repo": "govoplan-core",
+                "status": "planned",
+            },
+        ],
+        "notes": [],
+    }
+
+
+def mutating_first_plan() -> dict[str, object]:
+    plan = release_plan()
+    plan["dry_run_steps"] = [plan["dry_run_steps"][1]]  # type: ignore[index]
+    return plan
+
+
+if __name__ == "__main__":
+    unittest.main()

tools/release/server/app.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import hashlib
 from pathlib import Path
 
 from fastapi import FastAPI, HTTPException, Request
@@ -21,6 +22,12 @@ from govoplan_release import (
     tag_repositories,
 )
 from govoplan_release.model import to_jsonable
+from govoplan_release.release_run import (
+    ReleaseRunConflict,
+    ReleaseRunCorrupt,
+    ReleaseRunNotFound,
+    ReleaseRunStore,
+)
 from govoplan_release.workspace import DEFAULT_WORKSPACE_ROOT, META_ROOT
 
 
@@ -90,10 +97,35 @@ class RepositoryTagRequest(BaseModel):
     confirm: str = ""
 
 
-def create_app(*, workspace_root: Path = DEFAULT_WORKSPACE_ROOT, token: str | None = None) -> FastAPI:
+class ReleaseRunCreateRequest(BaseModel):
+    repo_versions: dict[str, str] = Field(default_factory=dict)
+    channel: str = Field(default="stable", min_length=1, max_length=64)
+    online: bool = False
+    remote_tags: bool = False
+    public_catalog: bool = True
+    include_migrations: bool = False
+
+
+class ReleaseRunCommandRequest(BaseModel):
+    request_id: str = Field(min_length=8, max_length=128)
+    reconciled: bool = False
+
+
+def create_app(
+    *,
+    workspace_root: Path = DEFAULT_WORKSPACE_ROOT,
+    token: str | None = None,
+    run_state_root: Path | None = None,
+) -> FastAPI:
     app = FastAPI(title="GovOPlaN Release Console", version="0.1.0")
     app.state.workspace_root = workspace_root
     app.state.token = token
+    app.state.release_runs = ReleaseRunStore(
+        run_state_root
+        if run_state_root is not None
+        else default_release_run_root(workspace_root)
+    )
+    app.state.workspace_fingerprint = release_workspace_fingerprint(workspace_root)
 
     @app.middleware("http")
     async def require_token(request: Request, call_next):  # type: ignore[no-untyped-def]
@@ -202,6 +234,105 @@ def create_app(*, workspace_root: Path = DEFAULT_WORKSPACE_ROOT, token: str | No
         )
         return to_jsonable(release_plan)
 
+    @app.get("/api/release-runs")
+    def release_runs(limit: int = 20) -> dict[str, object]:
+        return {"runs": app.state.release_runs.list(limit=limit)}
+
+    @app.post("/api/release-runs")
+    def create_release_run(request: ReleaseRunCreateRequest) -> dict[str, object]:
+        if not request.repo_versions:
+            raise HTTPException(
+                status_code=400, detail="At least one repository version is required"
+            )
+        snapshot = build_dashboard(
+            workspace_root=app.state.workspace_root,
+            online=request.online,
+            check_remote_tags=request.online or request.remote_tags,
+            check_public_catalog=request.public_catalog,
+            include_migrations=request.include_migrations,
+            channel=request.channel,
+        )
+        plan = build_selective_release_plan(
+            snapshot,
+            selected_repos=tuple(request.repo_versions),
+            repo_versions=dict(request.repo_versions),
+            channel=request.channel,
+        )
+        plan_payload = to_jsonable(plan)
+        planned_units = plan_payload.get("units")
+        planned_versions = (
+            {
+                unit.get("repo"): unit.get("target_version")
+                for unit in planned_units
+                if isinstance(unit, dict)
+                and isinstance(unit.get("repo"), str)
+                and isinstance(unit.get("target_version"), str)
+            }
+            if isinstance(planned_units, list)
+            else {}
+        )
+        if planned_versions != dict(request.repo_versions) or len(
+            planned_versions
+        ) != len(planned_units or []):
+            raise HTTPException(
+                status_code=409,
+                detail=(
+                    "The release plan did not resolve exactly the requested repository "
+                    "versions; refresh the dashboard and select known release units."
+                ),
+            )
+        try:
+            return app.state.release_runs.create(
+                input_snapshot={
+                    "channel": request.channel,
+                    "repo_versions": dict(request.repo_versions),
+                    "online": request.online,
+                    "remote_tags": request.remote_tags,
+                    "public_catalog": request.public_catalog,
+                    "include_migrations": request.include_migrations,
+                    "workspace_fingerprint": app.state.workspace_fingerprint,
+                },
+                plan_snapshot=plan_payload,
+            )
+        except (ReleaseRunConflict, ReleaseRunCorrupt) as exc:
+            raise HTTPException(status_code=409, detail=str(exc)) from exc
+
+    @app.get("/api/release-runs/{run_id}")
+    def release_run(run_id: str) -> dict[str, object]:
+        try:
+            return app.state.release_runs.get(run_id)
+        except ReleaseRunNotFound as exc:
+            raise HTTPException(status_code=404, detail=str(exc)) from exc
+        except ReleaseRunCorrupt as exc:
+            raise HTTPException(status_code=409, detail=str(exc)) from exc
+
+    @app.post("/api/release-runs/{run_id}/resume")
+    def resume_release_run(
+        run_id: str, request: ReleaseRunCommandRequest
+    ) -> dict[str, object]:
+        try:
+            return app.state.release_runs.resume(run_id, request_id=request.request_id)
+        except ReleaseRunNotFound as exc:
+            raise HTTPException(status_code=404, detail=str(exc)) from exc
+        except (ReleaseRunConflict, ReleaseRunCorrupt) as exc:
+            raise HTTPException(status_code=409, detail=str(exc)) from exc
+
+    @app.post("/api/release-runs/{run_id}/steps/{step_id}/retry")
+    def retry_release_run_step(
+        run_id: str, step_id: str, request: ReleaseRunCommandRequest
+    ) -> dict[str, object]:
+        try:
+            return app.state.release_runs.retry_step(
+                run_id,
+                step_id,
+                request_id=request.request_id,
+                reconciled=request.reconciled,
+            )
+        except ReleaseRunNotFound as exc:
+            raise HTTPException(status_code=404, detail=str(exc)) from exc
+        except (ReleaseRunConflict, ReleaseRunCorrupt) as exc:
+            raise HTTPException(status_code=409, detail=str(exc)) from exc
+
     @app.get("/api/catalog-candidates")
     def catalog_candidates() -> dict[str, object]:
         return {"candidates": list_catalog_candidates()}
@@ -319,6 +450,22 @@ def create_app(*, workspace_root: Path = DEFAULT_WORKSPACE_ROOT, token: str | No
     return app
 
 
+def default_release_run_root(workspace_root: Path) -> Path:
+    workspace = workspace_root.expanduser().resolve()
+    if (workspace / "repositories.json").is_file():
+        return workspace / "runtime" / "release-runs"
+    workspace_meta = workspace / "govoplan"
+    if (workspace_meta / "repositories.json").is_file():
+        return workspace_meta / "runtime" / "release-runs"
+    fingerprint = release_workspace_fingerprint(workspace)
+    return META_ROOT / "runtime" / "release-runs" / f"workspace-{fingerprint[:16]}"
+
+
+def release_workspace_fingerprint(workspace_root: Path) -> str:
+    canonical = str(workspace_root.expanduser().resolve()).encode("utf-8")
+    return hashlib.sha256(canonical).hexdigest()
+
+
 def parse_csv(value: str | None) -> tuple[str, ...]:
     if not value:
         return ()

tools/release/webui/index.html

@@ -96,7 +96,8 @@
       }
 
       input[type="text"],
-      input[type="password"] {
+      input[type="password"],
+      select {
         width: 100%;
         min-height: 36px;
         border: 1px solid var(--line);
@@ -139,7 +140,7 @@
       }
 
       button:disabled {
-        cursor: wait;
+        cursor: not-allowed;
         opacity: 0.65;
       }
 
@@ -635,6 +636,31 @@
         </div>
 
         <div class="side">
+          <section>
+            <div class="section-head">
+              <h2>Durable Run State</h2>
+              <small id="runStatus">no run selected</small>
+            </div>
+            <div class="details">
+              <p class="hint"><strong>State tracking only.</strong> A run freezes the selected inputs and plan, guides recovery, and records bounded state events. It does not execute or confirm tags, signing, publication, or catalog changes.</p>
+              <div class="form-grid">
+                <div>
+                  <label for="releaseRun">Saved run</label>
+                  <select id="releaseRun">
+                    <option value="">No saved run selected</option>
+                  </select>
+                </div>
+              </div>
+              <div class="button-row">
+                <button id="createReleaseRun">Create Run from Selection</button>
+                <button class="secondary" id="resumeReleaseRun" disabled>Resume Run</button>
+              </div>
+              <div class="actions" id="runOutput">
+                <div class="muted">Create a run after selecting exact repository versions, or choose a saved run.</div>
+              </div>
+            </div>
+          </section>
+
           <section>
             <div class="section-head">
               <h2>Release Workflow</h2>
@@ -857,6 +883,11 @@
         previewPrepare: document.getElementById("previewPrepare"),
         commitPrepare: document.getElementById("commitPrepare"),
         buildSelectedPlan: document.getElementById("buildSelectedPlan"),
+        releaseRun: document.getElementById("releaseRun"),
+        runStatus: document.getElementById("runStatus"),
+        runOutput: document.getElementById("runOutput"),
+        createReleaseRun: document.getElementById("createReleaseRun"),
+        resumeReleaseRun: document.getElementById("resumeReleaseRun"),
         selectUnpushedUnits: document.getElementById("selectUnpushedUnits"),
         selectChangedUnits: document.getElementById("selectChangedUnits"),
         selectUnreleasedUnits: document.getElementById("selectUnreleasedUnits"),
@@ -874,6 +905,8 @@
         rows: {},
         dashboard: null,
         manualRepo: "",
+        currentRun: null,
+        runs: [],
       };
 
       elements.refresh.addEventListener("click", load);
@@ -889,6 +922,9 @@
       elements.previewPrepare.addEventListener("click", () => prepareSelectedRepos({ apply: false }));
       elements.commitPrepare.addEventListener("click", () => prepareSelectedRepos({ apply: true }));
       elements.buildSelectedPlan.addEventListener("click", buildSelectedPlan);
+      elements.createReleaseRun.addEventListener("click", createReleaseRun);
+      elements.resumeReleaseRun.addEventListener("click", resumeReleaseRun);
+      elements.releaseRun.addEventListener("change", () => loadReleaseRun(elements.releaseRun.value));
       elements.previewReleaseTags.addEventListener("click", () => releaseSelectedTags({ apply: false, push: true }));
       elements.createReleaseTags.addEventListener("click", () => releaseSelectedTags({ apply: true, push: false }));
       elements.publishReleaseTags.addEventListener("click", () => releaseSelectedTags({ apply: true, push: true }));
@@ -949,10 +985,15 @@
       async function load() {
         elements.statusLine.textContent = "Refreshing...";
         try {
-          const [dashboard, intelligence] = await Promise.all([api("/api/dashboard"), api("/api/release-intelligence")]);
+          const [dashboard, intelligence, runs] = await Promise.all([api("/api/dashboard"), api("/api/release-intelligence"), api("/api/release-runs")]);
           renderDashboard(dashboard);
           renderReleaseIntelligence(intelligence);
+          renderReleaseRunList(runs.runs || []);
+          if (releaseState.currentRun?.run_id) {
+            await loadReleaseRun(releaseState.currentRun.run_id, { restoreSelection: false });
+          } else {
             renderIdlePlan();
+          }
           elements.statusLine.textContent = `Generated ${dashboard.generated_at}`;
         } catch (error) {
           elements.statusLine.innerHTML = `<span class="error">${escapeHtml(error.message)}</span>`;
@@ -977,6 +1018,156 @@
         renderCatalog(data.catalog, data.target_version);
       }
 
+      function renderReleaseRunList(runs) {
+        releaseState.runs = Array.isArray(runs) ? runs : [];
+        const selectedId = releaseState.currentRun?.run_id || "";
+        const options = releaseState.runs.map((run) => {
+          const repositories = run.repo_versions ? Object.keys(run.repo_versions).length : 0;
+          const label = `${run.run_id} · ${run.status || "unknown"} · ${repositories} repos`;
+          return `<option value="${escapeAttr(run.run_id)}" ${run.run_id === selectedId ? "selected" : ""}>${escapeHtml(label)}</option>`;
+        }).join("");
+        elements.releaseRun.innerHTML = `<option value="">No saved run selected</option>${options}`;
+        if (selectedId && releaseState.runs.some((run) => run.run_id === selectedId)) {
+          elements.releaseRun.value = selectedId;
+        }
+      }
+
+      async function createReleaseRun() {
+        const repoVersions = selectedRepoVersions();
+        if (!Object.keys(repoVersions).length) {
+          elements.runStatus.textContent = "selection required";
+          elements.runOutput.innerHTML = `<div class="action"><h3>${pill("unavailable", "warn")} Select release units</h3><p>Select one or more repositories and exact target versions before creating a durable run.</p></div>`;
+          return;
+        }
+        setBusy([elements.createReleaseRun], true);
+        elements.runStatus.textContent = "freezing plan...";
+        try {
+          const run = await postJson("/api/release-runs", {
+            repo_versions: repoVersions,
+            channel: elements.channel.value.trim() || "stable",
+            online: false,
+            remote_tags: elements.online.checked,
+            public_catalog: elements.publicCatalog.checked,
+            include_migrations: elements.migrations.checked,
+          });
+          releaseState.currentRun = run;
+          restoreReleaseRunSelection(run);
+          renderReleaseRun(run);
+          const runs = await api("/api/release-runs");
+          renderReleaseRunList(runs.runs || []);
+          elements.releaseRun.value = run.run_id;
+        } catch (error) {
+          elements.runStatus.textContent = "error";
+          elements.runOutput.innerHTML = `<div class="action"><h3>${pill("error", "block")} Run creation failed</h3><p>${escapeHtml(error.message)}</p></div>`;
+        } finally {
+          setBusy([elements.createReleaseRun], false);
+        }
+      }
+
+      async function loadReleaseRun(runId, options = {}) {
+        if (!runId) {
+          releaseState.currentRun = null;
+          elements.runStatus.textContent = "no run selected";
+          elements.resumeReleaseRun.disabled = true;
+          elements.runOutput.innerHTML = `<div class="muted">Create a run after selecting exact repository versions, or choose a saved run.</div>`;
+          return;
+        }
+        elements.runStatus.textContent = "loading...";
+        try {
+          const run = await api(`/api/release-runs/${encodeURIComponent(runId)}`);
+          releaseState.currentRun = run;
+          if (options.restoreSelection !== false) restoreReleaseRunSelection(run);
+          renderReleaseRun(run);
+          elements.releaseRun.value = run.run_id;
+        } catch (error) {
+          releaseState.currentRun = null;
+          elements.runStatus.textContent = "unavailable";
+          elements.resumeReleaseRun.disabled = true;
+          elements.runOutput.innerHTML = `<div class="action"><h3>${pill("unavailable", "block")} Run cannot be read</h3><p>${escapeHtml(error.message)}</p></div>`;
+        }
+      }
+
+      function restoreReleaseRunSelection(run) {
+        const input = run.immutable?.input || {};
+        const repoVersions = input.repo_versions || {};
+        elements.channel.value = input.channel || "stable";
+        elements.publicCatalog.checked = input.public_catalog === true;
+        elements.online.checked = input.remote_tags === true || input.online === true;
+        elements.migrations.checked = input.include_migrations === true;
+        for (const repo of Object.keys(releaseState.rows)) {
+          releaseState.rows[repo] = {
+            ...releaseState.rows[repo],
+            selected: Object.prototype.hasOwnProperty.call(repoVersions, repo),
+            ...(Object.prototype.hasOwnProperty.call(repoVersions, repo) ? { target: repoVersions[repo] } : {}),
+          };
+        }
+        if (releaseState.dashboard) renderReleaseUnits(releaseState.dashboard);
+        if (run.immutable?.plan) renderPlan(run.immutable.plan);
+      }
+
+      function renderReleaseRun(run) {
+        const state = run.state || {};
+        const recommendation = run.recommended_next || {};
+        const steps = Array.isArray(state.steps) ? state.steps : [];
+        const events = Array.isArray(state.events) ? state.events.slice(-5).reverse() : [];
+        const recommendationKind = state.status === "blocked" ? "block" : recommendation.available ? "ok" : "warn";
+        const recommendationHtml = recommendation.id ? `<div class="action recommended">
+          <h3>${pill("recommended next", recommendationKind)} ${escapeHtml(recommendation.title || recommendation.id)}</h3>
+          <p>${escapeHtml(recommendation.detail || "")}</p>
+          <p class="action-meta"><strong>How to continue:</strong> ${escapeHtml(recommendation.remediation || "")}</p>
+        </div>` : "";
+        const stepHtml = steps.map((step) => {
+          const kind = step.state === "succeeded" ? "ok" : ["failed", "interrupted"].includes(step.state) ? "block" : step.available ? "ok" : "warn";
+          const retryReason = step.retry_available ? "Prepare this known failed or read-only interrupted step for an explicit retry." : (step.disabled_reason || "Retry is unavailable for this state.");
+          return `<div class="action">
+            <div class="stage-title"><h3>${escapeHtml(`${step.order}. ${step.title || step.id}`)}</h3>${pill(step.state, kind)}</div>
+            <p>${escapeHtml(step.detail || "")}</p>
+            ${step.result_code ? `<p class="action-meta">Result code: ${escapeHtml(step.result_code)}</p>` : ""}
+            <div class="button-row"><button class="secondary" data-run-retry-step="${escapeAttr(step.id)}" ${step.retry_available ? "" : "disabled"} title="${escapeAttr(retryReason)}">Prepare Retry</button></div>
+          </div>`;
+        }).join("");
+        const eventHtml = events.length ? `<div class="action"><h3>Recent bounded state events</h3>${events.map((event) => `<p class="action-meta">${escapeHtml(`${event.at} · ${event.type}${event.step_id ? ` · ${event.step_id}` : ""}${event.result_code ? ` · ${event.result_code}` : ""}`)}</p>`).join("")}</div>` : "";
+        elements.runStatus.textContent = state.status || "unknown";
+        elements.resumeReleaseRun.disabled = state.status === "completed";
+        elements.runOutput.innerHTML = `<div class="action"><h3>${pill(state.status || "unknown", state.status === "blocked" ? "block" : state.status === "completed" ? "ok" : "warn")} ${escapeHtml(run.run_id)}</h3><p>Frozen plan ${escapeHtml((run.immutable?.digest || "").slice(0, 16))}… · updated ${escapeHtml(run.updated_at || "-")}</p><p class="action-meta">Executor controls remain separate and keep their existing confirmations.</p></div>${recommendationHtml}${stepHtml}${eventHtml}`;
+        for (const button of elements.runOutput.querySelectorAll("[data-run-retry-step]")) {
+          button.addEventListener("click", () => retryReleaseRunStep(button.dataset.runRetryStep));
+        }
+      }
+
+      async function resumeReleaseRun() {
+        const run = releaseState.currentRun;
+        if (!run) return;
+        setBusy([elements.resumeReleaseRun], true);
+        try {
+          const resumed = await postJson(`/api/release-runs/${encodeURIComponent(run.run_id)}/resume`, { request_id: requestId("resume") });
+          releaseState.currentRun = resumed;
+          renderReleaseRun(resumed);
+        } catch (error) {
+          elements.runStatus.textContent = "error";
+          elements.runOutput.insertAdjacentHTML("afterbegin", `<div class="action"><h3>${pill("error", "block")} Resume failed</h3><p>${escapeHtml(error.message)}</p></div>`);
+        } finally {
+          elements.resumeReleaseRun.disabled = releaseState.currentRun?.state?.status === "completed";
+        }
+      }
+
+      async function retryReleaseRunStep(stepId) {
+        const run = releaseState.currentRun;
+        if (!run || !stepId) return;
+        try {
+          const retried = await postJson(`/api/release-runs/${encodeURIComponent(run.run_id)}/steps/${encodeURIComponent(stepId)}/retry`, { request_id: requestId("retry"), reconciled: false });
+          releaseState.currentRun = retried;
+          renderReleaseRun(retried);
+        } catch (error) {
+          elements.runOutput.insertAdjacentHTML("afterbegin", `<div class="action"><h3>${pill("error", "block")} Retry preparation failed</h3><p>${escapeHtml(error.message)}</p></div>`);
+        }
+      }
+
+      function requestId(prefix) {
+        const identifier = window.crypto?.randomUUID ? window.crypto.randomUUID() : `${Date.now()}-${Math.random().toString(16).slice(2)}`;
+        return `${prefix}-${identifier}`;
+      }
+
       function renderWorkflowStages(data) {
         const s = data.summary || {};
         const selected = elements.releaseUnits ? elements.releaseUnits.querySelectorAll("[data-release-check]:checked").length : 0;