Prove that pagination finished before publishing a dataset

Read the provider’s actual pagination model

World Bank Indicators V2 documents page and per_page parameters, while FRED V1 observations uses limit and offset. These are different traversal conventions. An adapter should declare which one it supports and validate the accompanying metadata rather than applying a generic next-page guess to every provider.

Freeze the other query parameters for the entire run: selected series or indicators, countries, periods, output mode and vintage bounds where supported. A page request with a changed filter belongs to a different dataset. Record a canonical request fingerprint so a resumed run cannot accidentally combine pages from two different queries.

Bind pages to the same safe request signature: series, countries, period range, filters and output mode. Matching counts cannot repair a run whose second page accidentally uses a different indicator or year range.

Reconcile a synthetic three-page response

Suppose a synthetic page envelope reports total 125 and page size 50. Expected page count is ceiling(125 / 50) = 3. The fixture supplies 50 records on page 1, 50 on page 2 and 25 on page 3. Accepted raw count should total 125 before any documented filtering.

Now make page 2 contain one repeated identity from page 1. Raw count still reaches 125, but unique count becomes 124. A total-count check alone passes while completeness remains questionable. Report duplicate identities and investigate whether they are legitimate revisions, repeated observations or an inconsistent page sequence before accepting the dataset.

Do not assume a mutable API is a snapshot

If upstream records change between page requests, offset pagination can skip or repeat items as ordering shifts. A provider-supplied snapshot token or fixed vintage query can help where documented, but the adapter must not claim snapshot consistency merely because all pages returned successfully.

For a source without a snapshot guarantee, record the run’s start and end retrieval times and inspect whether page totals or metadata change mid-run. A mismatch should trigger a bounded restart or an incomplete status under the chosen policy. Retrying indefinitely until totals happen to agree does not establish a historical atomic snapshot.

Handle empty and malformed pages explicitly

An empty page before the declared final page is not automatically the end of the dataset. It may be a source error, a changed result set or a malformed response. Validate the envelope, page number, totals and record-array type before deciding how to proceed.

Use synthetic failures to test a missing middle page, a repeated page number, contradictory totals and an error object delivered where records were expected. Preserve diagnostic response metadata without leaking credentials. The ingestion output should distinguish no records for a valid query from records unavailable because the request failed.

Publish only a reconciled generation

Stage pages under a run identifier, normalize them, check schema and identity rules, then commit a completion manifest containing expected and accepted counts. Downstream consumers should read the last complete generation rather than a directory that fills up one page at a time.

The fixture workbench demonstrates parsing and reconciliation for selected response shapes. It does not operate a live paginator or guarantee that current provider datasets remain static during collection. A production implementation needs bounded request policies, durable cloud storage and recovery tests, with any unresolved consistency limitation carried into the dataset’s provenance rather than hidden behind a green download message.

Keep in the decision record

• Freeze the query across all pages.
• Reconcile raw count, unique identity and page metadata.
• Publish a completed generation, never an accidental partial prefix.

Source notes

1. World Bank — Indicators API basic call structures
2. FRED API V1 — Series observations
3. World Bank — API Error Codes

Primary specifications support the interface descriptions. All downloadable response fixtures are synthetic. Access and rights checks are dated in the source inventory.

Continue the analysis