fix(release): harden durable run recovery
This commit is contained in:
@@ -63,38 +63,47 @@ these digests detect accidental or manual corruption, not an attacker who can
|
||||
replace the private record and recompute its checksums. Changing a target,
|
||||
channel, or gate input requires a new run.
|
||||
|
||||
Run records live below `runtime/release-runs/`. They survive console restarts,
|
||||
but remain local operator state rather than a signed release artifact or the
|
||||
system audit log. The directory is restricted to mode `0700` and records to
|
||||
`0600`. Writes use a cross-process lock, a same-directory temporary file,
|
||||
`fsync`, atomic replacement, and directory `fsync`. Symbolic-link storage paths,
|
||||
overly broad record modes, malformed schemas, unknown fields, invalid state
|
||||
combinations, oversized files, and digest mismatches fail closed. A bad record
|
||||
is neither rewritten nor automatically quarantined.
|
||||
Run records survive console restarts, but remain local operator state rather
|
||||
than a signed release artifact or the system audit log. The default location is
|
||||
`$XDG_STATE_HOME/govoplan/release-console/workspace-<sha256>/release-runs/`, or
|
||||
`~/.local/state/...` when `XDG_STATE_HOME` is unset or relative. It is outside
|
||||
the source checkout so filesystems without enforceable POSIX modes cannot
|
||||
silently weaken the journal. Newly created state directories use mode `0700`
|
||||
and records use `0600`. Writes use a cross-process lock, a same-directory
|
||||
temporary file, `fsync`, atomic replacement, and directory/parent `fsync`.
|
||||
Symbolic-link paths, untrusted owners or writable ancestry, overly broad record
|
||||
modes, malformed schemas, unknown fields, invalid state combinations,
|
||||
oversized files, and digest mismatches fail closed. A bad record is neither
|
||||
rewritten nor automatically quarantined.
|
||||
|
||||
The default store belongs to the selected workspace: the console uses that
|
||||
workspace's meta checkout when it is available, or a stable workspace-digest
|
||||
namespace below this checkout's runtime directory otherwise. The same digest is
|
||||
part of the immutable input snapshot. An alternate workspace therefore cannot
|
||||
silently list or resume another workspace's runs. `run_state_root` remains an
|
||||
explicit embedding/test override.
|
||||
The full resolved-workspace SHA-256 is part of both the private storage
|
||||
namespace and immutable input snapshot. Every list, read, and state transition
|
||||
checks it. An alternate workspace therefore cannot list or resume another
|
||||
workspace's runs even when an embedding/test `run_state_root` override is
|
||||
shared accidentally.
|
||||
|
||||
Each frozen plan step has an explicit `pending`, `running`, `succeeded`,
|
||||
`failed`, or `interrupted` state. Plan order remains a prerequisite: a later
|
||||
step is unavailable until earlier steps have succeeded. Exact attempt and
|
||||
resume/retry request identifiers are fingerprinted so repeated state commands
|
||||
are idempotent. The record keeps at most 256 server-generated state events and
|
||||
128 request fingerprints. Events contain only an enum event type, timestamp,
|
||||
step identifier, and bounded result code—never commands, process output,
|
||||
confirmation text, credentials, bearer tokens, or signing material.
|
||||
resume/retry/reconciliation request identifiers are fingerprinted so delayed
|
||||
repetitions remain idempotent for the run's lifetime. These fingerprints are
|
||||
never evicted: the store fails closed before accepting more than 2,048 commands
|
||||
or 2,048 attempts and asks the operator to create a fresh run. The display
|
||||
record keeps at most 256 server-generated state events. Events contain only an
|
||||
enum event type, timestamp, step identifier, and bounded result code—never
|
||||
commands, process output, confirmation text, credentials, bearer tokens, or
|
||||
signing material.
|
||||
|
||||
An explicit resume after a process restart converts every persisted `running`
|
||||
step to `interrupted`; it never guesses whether an external effect happened.
|
||||
An interrupted read-only step can be prepared for retry. An interrupted
|
||||
mutating step remains unavailable until the operator has reconciled local and
|
||||
remote state and explicitly records that reconciliation when preparing the
|
||||
retry. A known failed attempt can likewise be prepared for retry. The UI keeps
|
||||
the retry control visible and explains why it is disabled when retry is unsafe.
|
||||
mutating step remains unavailable until the operator independently reconciles
|
||||
local and remote state, selects `effect_absent`, `effect_succeeded`, or
|
||||
`unresolved`, and types `RECONCILE`. `effect_absent` prepares a safe new
|
||||
attempt, `effect_succeeded` advances the run without repeating the external
|
||||
effect, and `unresolved` keeps the run blocked. Each outcome emits a bounded,
|
||||
code-only state event. A known failed attempt can likewise be prepared for
|
||||
retry. The UI keeps unavailable controls visible and disabled.
|
||||
|
||||
The run API is covered by the same local console token middleware as every
|
||||
other `/api/` route:
|
||||
@@ -105,7 +114,12 @@ other `/api/` route:
|
||||
- `GET /api/release-runs/{run_id}` reads and verifies one exact record.
|
||||
- `POST /api/release-runs/{run_id}/resume` records explicit recovery.
|
||||
- `POST /api/release-runs/{run_id}/steps/{step_id}/retry` prepares a failed or
|
||||
safely reconciled interrupted step for another attempt.
|
||||
read-only interrupted step for another attempt.
|
||||
- `POST /api/release-runs/{run_id}/steps/{step_id}/reconcile` records a
|
||||
confirmed observed outcome for an interrupted mutating step.
|
||||
|
||||
Run-storage errors are confined to the Durable Run State card; dashboard and
|
||||
release-preview collection continue and show the bounded storage remediation.
|
||||
|
||||
This foundation tracks state only. It deliberately does not run a plan step or
|
||||
infer success from a button click. Existing preview and executor controls stay
|
||||
|
||||
Reference in New Issue
Block a user