Make replay safe and commit checkpoints atomically

HTTP idempotency is not a complete pipeline guarantee

RFC 9110 describes idempotent method semantics in terms of the intended effect on the server. Repeating a GET can be appropriate at that layer, but the returned representation can change over time. The client can also duplicate rows or lose work even when its requests use an idempotent method.

Define a separate ingestion invariant: replaying the same retained raw evidence under the same mapping version produces the same normalized identities and content. Use a run identifier for attempts, observation identity for measured items and revision_key for versions. Do not confuse any of those with a random database row identifier.

Walk through a crash boundary

Consider a synthetic batch containing records A, B and C with a next-page checkpoint of 2. If the collector writes checkpoint 2 before storing C and then crashes, a restart may skip C permanently. If it stores A, B and C but crashes before the checkpoint, a restart may fetch the batch again.

The second ordering can be recovered safely with idempotent writes; the first loses data unless additional reconciliation detects it. A stronger design commits the accepted rows and checkpoint together in a transaction, or publishes an immutable batch manifest only after every referenced object is durable. The storage mechanism determines the exact implementation.

Define duplicate and conflict behavior

If a replay has the same revision key and the same semantic content, it should be recognized as a repeat rather than appended as a new observation. If the same supposedly unique revision key arrives with different content, report a conflict. Overwriting it silently destroys evidence about a violated assumption.

A synthetic test can ingest A=10 and B=20 twice and expect two accepted identities, not four. Then replay B=21 under the same revision key and expect a conflict. Finally supply B=21 with documented new revision evidence and expect two versions of B, with selection governed by the revision policy.

Publish generations instead of mutable partial tables

For file-based storage, a practical pattern is to write a new generation in isolation, validate its manifest and then update a small pointer to the accepted generation. Consumers read the pointer and its manifest, not arbitrary files still being written. The pointer update needs the atomicity supported by the chosen storage service.

A manifest can contain schema version, mapping version, input digests, record count and completion status. If a referenced object is absent or has a mismatched digest, the generation is not complete. A completion flag set by a process that merely started is not durable evidence that ingestion finished successfully.

Test recovery at every meaningful transition

Inject synthetic failure after raw capture, after normalization, during row persistence, before checkpoint commit and after commit but before acknowledgment. Restart and verify either safe replay or recognition of completed work. This tests the boundaries where a happy-path adapter demonstration provides little assurance.

These are design criteria for production integration, not a claim that the browser fixture lab contains a transactional database or live scheduler. Any real recurring collector must keep its state and execution in managed cloud infrastructure. Preserve recovery receipts and incomplete runs so an operator can distinguish successful replay from a process that simply stopped reporting errors.

Keep in the decision record

• Separate request-method semantics from client-side replay guarantees.
• Commit data and progress together, or publish a verified immutable generation.
• Treat same-key, different-content records as conflicts.

Source notes

1. RFC 9110 — HTTP Semantics, idempotent methods
2. FRED API V1 — Series vintage dates
3. JSON Schema — Object reference

Primary specifications support the interface descriptions. All downloadable response fixtures are synthetic. Access and rights checks are dated in the source inventory.

Continue the analysis