govoplan-campaign/docs/CAMPAIGN_BUILD_RECOVERY.md

Campaign Build Recovery

Campaign build is fenced per tenant and Campaign version. Before rendering or writing generated artifacts, Campaign commits a Core recovery operation with the canonical source and validation hashes, the runtime fence, and a reserved opaque object prefix. A repeated idempotency key can replay only a verified successful build; it cannot start a second active build.

Generated EML and bounded print output are written to shared storage and checked for exact size and SHA-256 content. Campaign renews the build fence after that check and before changing jobs, then commits its jobs and execution snapshot, compares the stored object and database manifests, and records verified success. A managed Files output makes the operation forward-recoverable because Campaign cannot undo a Files-owned artifact; object-only builds use explicit compensation.

If the Campaign database transaction fails, Campaign deletes every object it recorded and verifies absence before recording recovered state. Failed deletion, an unavailable storage check, process loss, or superseded-object cleanup failure leaves a recovery-required operation visible in Ops. Do not retry such an operation as a normal build. Verify its checkpoint chain and reserved prefix, then reconcile it through the owning-module procedure.

Orphan inventory and cleanup

The operator-only endpoint POST /api/v1/campaigns/operations/artifacts/reconcile requires system:settings:write. It never scans outside campaign-artifacts/{tenant_id}/, and one request reads at most page_size objects. The default request is a dry run:

{}

The response reports each eligible key, size, modification time, age, reason, page totals, and next_cursor. Continue with that cursor to inspect the next bounded page. Objects are not eligible when they are referenced by a committed EML or print-output row, belong to an actively fenced build, have no trustworthy modification time, have an invalid build-key shape, or are younger than the grace period. The minimum and default grace period is 24 hours.

Apply an inspected page with a new idempotency key:

{
  "apply": true,
  "idempotency_key": "incident-2026-08-03-page-1",
  "grace_period_hours": 24,
  "page_size": 250
}

An applied run rechecks committed references and active build leases before each bounded deletion batch. A Core distributed lease prevents two nodes from committing the same tenant cleanup concurrently. Every attempted deletion is probed afterward. recovery_required means an object was verified to remain; outcome_unknown means storage could not prove whether the delete took effect. Use a new idempotency key to retry after the storage problem is corrected. A successful repeated request with the same key returns already_completed and does not delete again.

Cleanup never clears Campaign database references. Audit and recovery evidence records counts and hashed manifests rather than object keys. Exact keys are returned only by this privileged endpoint and must stay in restricted incident records.

No recovery checkpoint contains message bodies, recipients, credentials, or resolved provider secrets. Object keys remain restricted diagnostics rather than Campaign business data.

Database rows, campaign-artifacts/ objects, and the encryption/key service are one coordinated backup and recovery boundary. After any partial restore, pause delivery, run a dry inventory, reconcile Campaign recovery operations in Ops, and verify referenced object hashes before workers resume.