Reconcile orphaned Campaign artifacts

docs/CAMPAIGN_BUILD_RECOVERY.md

@@ -7,20 +7,69 @@ opaque object prefix. A repeated idempotency key can replay only a verified
 successful build; it cannot start a second active build.
 
 Generated EML and bounded print output are written to shared storage and checked
-for exact size and SHA-256 content. Campaign then commits its jobs and execution
-snapshot, compares the stored object and database manifests, and records
-verified success. A managed Files output makes the operation forward-recoverable
-because Campaign cannot undo a Files-owned artifact; object-only builds use
-explicit compensation.
+for exact size and SHA-256 content. Campaign renews the build fence after that
+check and before changing jobs, then commits its jobs and execution snapshot,
+compares the stored object and database manifests, and records verified success.
+A managed Files output makes the operation forward-recoverable because Campaign
+cannot undo a Files-owned artifact; object-only builds use explicit
+compensation.
 
 If the Campaign database transaction fails, Campaign deletes every object it
 recorded and verifies absence before recording recovered state. Failed deletion,
 an unavailable storage check, process loss, or superseded-object cleanup failure
 leaves a recovery-required operation visible in Ops. Do not retry such an
 operation as a normal build. Verify its checkpoint chain and reserved prefix,
-then reconcile it through the owning-module procedure. The orphan inventory
-reconciler tracked in Campaign issue 91 will automate that bounded inspection.
+then reconcile it through the owning-module procedure.
+
+## Orphan inventory and cleanup
+
+The operator-only endpoint
+`POST /api/v1/campaigns/operations/artifacts/reconcile` requires
+`system:settings:write`. It never scans outside
+`campaign-artifacts/{tenant_id}/`, and one request reads at most `page_size`
+objects. The default request is a dry run:
+
+```json
+{}
+```
+
+The response reports each eligible key, size, modification time, age, reason,
+page totals, and `next_cursor`. Continue with that cursor to inspect the next
+bounded page. Objects are not eligible when they are referenced by a committed
+EML or print-output row, belong to an actively fenced build, have no trustworthy
+modification time, have an invalid build-key shape, or are younger than the
+grace period. The minimum and default grace period is 24 hours.
+
+Apply an inspected page with a new idempotency key:
+
+```json
+{
+  "apply": true,
+  "idempotency_key": "incident-2026-08-03-page-1",
+  "grace_period_hours": 24,
+  "page_size": 250
+}
+```
+
+An applied run rechecks committed references and active build leases before
+each bounded deletion batch. A Core distributed lease prevents two nodes from
+committing the same tenant cleanup concurrently. Every attempted deletion is
+probed afterward. `recovery_required` means an object was verified to remain;
+`outcome_unknown` means storage could not prove whether the delete took effect.
+Use a new idempotency key to retry after the storage problem is corrected. A
+successful repeated request with the same key returns `already_completed` and
+does not delete again.
+
+Cleanup never clears Campaign database references. Audit and recovery evidence
+records counts and hashed manifests rather than object keys. Exact keys are
+returned only by this privileged endpoint and must stay in restricted incident
+records.
 
 No recovery checkpoint contains message bodies, recipients, credentials, or
 resolved provider secrets. Object keys remain restricted diagnostics rather
 than Campaign business data.
+
+Database rows, `campaign-artifacts/` objects, and the encryption/key service are
+one coordinated backup and recovery boundary. After any partial restore, pause
+delivery, run a dry inventory, reconcile Campaign recovery operations in Ops,
+and verify referenced object hashes before workers resume.

docs/CAMPAIGN_DELIVERY_RUNBOOK.md

@@ -169,8 +169,11 @@ before attempting delivery.
   leaves an outcome-unknown operation; a deletion/metadata mismatch becomes
   recovery-required in Ops.
 - A hard process loss between object creation and metadata commit can leave an
-  orphan object. Reconcile only within the Campaign build prefix and verify that
-  no job references the object before deleting it.
+  orphan object. Use the operator-only, dry-run-first Campaign artifact
+  reconciler documented in `CAMPAIGN_BUILD_RECOVERY.md`; it scans one bounded
+  tenant-prefix page, enforces a minimum 24-hour grace period, protects active
+  build fences, and rechecks committed EML and print-output references before
+  deletion.
 - Restore Campaign rows, object storage, and the encryption key to one
   coordinated recovery point before resuming workers.
 

docs/CAMPAIGN_HANDBOOK.md

@@ -397,6 +397,12 @@ After restore, keep outbound delivery paused until queue/attempt state and
 provider evidence have been reconciled; never let restored accepted jobs send
 again merely because a queue message was lost.
 
+For unreferenced generated objects after process loss, platform operators first
+run the bounded Campaign artifact inventory in dry-run mode. Apply only an
+inspected page with a unique incident idempotency key. The cleanup keeps exact
+keys out of ordinary Campaign responses, does not clear database references,
+and leaves storage failures in the Core recovery ledger for explicit retry.
+
 ### Incident handling
 
 1. Pause new delivery when duplicate or unknown effects are possible.