fix(release): harden durable run recovery

This commit is contained in:
2026-07-22 18:58:54 +02:00
parent ead697d049
commit 1dc9148ec3
5 changed files with 1046 additions and 127 deletions

View File

@@ -63,38 +63,47 @@ these digests detect accidental or manual corruption, not an attacker who can
replace the private record and recompute its checksums. Changing a target,
channel, or gate input requires a new run.
Run records live below `runtime/release-runs/`. They survive console restarts,
but remain local operator state rather than a signed release artifact or the
system audit log. The directory is restricted to mode `0700` and records to
`0600`. Writes use a cross-process lock, a same-directory temporary file,
`fsync`, atomic replacement, and directory `fsync`. Symbolic-link storage paths,
overly broad record modes, malformed schemas, unknown fields, invalid state
combinations, oversized files, and digest mismatches fail closed. A bad record
is neither rewritten nor automatically quarantined.
Run records survive console restarts, but remain local operator state rather
than a signed release artifact or the system audit log. The default location is
`$XDG_STATE_HOME/govoplan/release-console/workspace-<sha256>/release-runs/`, or
`~/.local/state/...` when `XDG_STATE_HOME` is unset or relative. It is outside
the source checkout so filesystems without enforceable POSIX modes cannot
silently weaken the journal. Newly created state directories use mode `0700`
and records use `0600`. Writes use a cross-process lock, a same-directory
temporary file, `fsync`, atomic replacement, and directory/parent `fsync`.
Symbolic-link paths, untrusted owners or writable ancestry, overly broad record
modes, malformed schemas, unknown fields, invalid state combinations,
oversized files, and digest mismatches fail closed. A bad record is neither
rewritten nor automatically quarantined.
The default store belongs to the selected workspace: the console uses that
workspace's meta checkout when it is available, or a stable workspace-digest
namespace below this checkout's runtime directory otherwise. The same digest is
part of the immutable input snapshot. An alternate workspace therefore cannot
silently list or resume another workspace's runs. `run_state_root` remains an
explicit embedding/test override.
The full resolved-workspace SHA-256 is part of both the private storage
namespace and immutable input snapshot. Every list, read, and state transition
checks it. An alternate workspace therefore cannot list or resume another
workspace's runs even when an embedding/test `run_state_root` override is
shared accidentally.
Each frozen plan step has an explicit `pending`, `running`, `succeeded`,
`failed`, or `interrupted` state. Plan order remains a prerequisite: a later
step is unavailable until earlier steps have succeeded. Exact attempt and
resume/retry request identifiers are fingerprinted so repeated state commands
are idempotent. The record keeps at most 256 server-generated state events and
128 request fingerprints. Events contain only an enum event type, timestamp,
step identifier, and bounded result code—never commands, process output,
confirmation text, credentials, bearer tokens, or signing material.
resume/retry/reconciliation request identifiers are fingerprinted so delayed
repetitions remain idempotent for the run's lifetime. These fingerprints are
never evicted: the store fails closed before accepting more than 2,048 commands
or 2,048 attempts and asks the operator to create a fresh run. The display
record keeps at most 256 server-generated state events. Events contain only an
enum event type, timestamp, step identifier, and bounded result code—never
commands, process output, confirmation text, credentials, bearer tokens, or
signing material.
An explicit resume after a process restart converts every persisted `running`
step to `interrupted`; it never guesses whether an external effect happened.
An interrupted read-only step can be prepared for retry. An interrupted
mutating step remains unavailable until the operator has reconciled local and
remote state and explicitly records that reconciliation when preparing the
retry. A known failed attempt can likewise be prepared for retry. The UI keeps
the retry control visible and explains why it is disabled when retry is unsafe.
mutating step remains unavailable until the operator independently reconciles
local and remote state, selects `effect_absent`, `effect_succeeded`, or
`unresolved`, and types `RECONCILE`. `effect_absent` prepares a safe new
attempt, `effect_succeeded` advances the run without repeating the external
effect, and `unresolved` keeps the run blocked. Each outcome emits a bounded,
code-only state event. A known failed attempt can likewise be prepared for
retry. The UI keeps unavailable controls visible and disabled.
The run API is covered by the same local console token middleware as every
other `/api/` route:
@@ -105,7 +114,12 @@ other `/api/` route:
- `GET /api/release-runs/{run_id}` reads and verifies one exact record.
- `POST /api/release-runs/{run_id}/resume` records explicit recovery.
- `POST /api/release-runs/{run_id}/steps/{step_id}/retry` prepares a failed or
safely reconciled interrupted step for another attempt.
read-only interrupted step for another attempt.
- `POST /api/release-runs/{run_id}/steps/{step_id}/reconcile` records a
confirmed observed outcome for an interrupted mutating step.
Run-storage errors are confined to the Durable Run State card; dashboard and
release-preview collection continue and show the bounded storage remediation.
This foundation tracks state only. It deliberately does not run a plan step or
infer success from a button click. Existing preview and executor controls stay