76 lines
3.5 KiB
Markdown
76 lines
3.5 KiB
Markdown
# Campaign Build Recovery
|
|
|
|
Campaign build is fenced per tenant and Campaign version. Before rendering or
|
|
writing generated artifacts, Campaign commits a Core recovery operation with
|
|
the canonical source and validation hashes, the runtime fence, and a reserved
|
|
opaque object prefix. A repeated idempotency key can replay only a verified
|
|
successful build; it cannot start a second active build.
|
|
|
|
Generated EML and bounded print output are written to shared storage and checked
|
|
for exact size and SHA-256 content. Campaign renews the build fence after that
|
|
check and before changing jobs, then commits its jobs and execution snapshot,
|
|
compares the stored object and database manifests, and records verified success.
|
|
A managed Files output makes the operation forward-recoverable because Campaign
|
|
cannot undo a Files-owned artifact; object-only builds use explicit
|
|
compensation.
|
|
|
|
If the Campaign database transaction fails, Campaign deletes every object it
|
|
recorded and verifies absence before recording recovered state. Failed deletion,
|
|
an unavailable storage check, process loss, or superseded-object cleanup failure
|
|
leaves a recovery-required operation visible in Ops. Do not retry such an
|
|
operation as a normal build. Verify its checkpoint chain and reserved prefix,
|
|
then reconcile it through the owning-module procedure.
|
|
|
|
## Orphan inventory and cleanup
|
|
|
|
The operator-only endpoint
|
|
`POST /api/v1/campaigns/operations/artifacts/reconcile` requires
|
|
`system:settings:write`. It never scans outside
|
|
`campaign-artifacts/{tenant_id}/`, and one request reads at most `page_size`
|
|
objects. The default request is a dry run:
|
|
|
|
```json
|
|
{}
|
|
```
|
|
|
|
The response reports each eligible key, size, modification time, age, reason,
|
|
page totals, and `next_cursor`. Continue with that cursor to inspect the next
|
|
bounded page. Objects are not eligible when they are referenced by a committed
|
|
EML or print-output row, belong to an actively fenced build, have no trustworthy
|
|
modification time, have an invalid build-key shape, or are younger than the
|
|
grace period. The minimum and default grace period is 24 hours.
|
|
|
|
Apply an inspected page with a new idempotency key:
|
|
|
|
```json
|
|
{
|
|
"apply": true,
|
|
"idempotency_key": "incident-2026-08-03-page-1",
|
|
"grace_period_hours": 24,
|
|
"page_size": 250
|
|
}
|
|
```
|
|
|
|
An applied run rechecks committed references and active build leases before
|
|
each bounded deletion batch. A Core distributed lease prevents two nodes from
|
|
committing the same tenant cleanup concurrently. Every attempted deletion is
|
|
probed afterward. `recovery_required` means an object was verified to remain;
|
|
`outcome_unknown` means storage could not prove whether the delete took effect.
|
|
Use a new idempotency key to retry after the storage problem is corrected. A
|
|
successful repeated request with the same key returns `already_completed` and
|
|
does not delete again.
|
|
|
|
Cleanup never clears Campaign database references. Audit and recovery evidence
|
|
records counts and hashed manifests rather than object keys. Exact keys are
|
|
returned only by this privileged endpoint and must stay in restricted incident
|
|
records.
|
|
|
|
No recovery checkpoint contains message bodies, recipients, credentials, or
|
|
resolved provider secrets. Object keys remain restricted diagnostics rather
|
|
than Campaign business data.
|
|
|
|
Database rows, `campaign-artifacts/` objects, and the encryption/key service are
|
|
one coordinated backup and recovery boundary. After any partial restore, pause
|
|
delivery, run a dry inventory, reconcile Campaign recovery operations in Ops,
|
|
and verify referenced object hashes before workers resume.
|