Retry failures within a bounded, provider-aware budget

Classify before repeating

A temporary connection failure can justify a retry. A malformed request, unsupported field or rejected credential usually requires correction rather than repetition. Treat status code, provider error body and parsing outcome as separate evidence. A successful connection carrying an invalid response is not repaired by labeling it a network problem.

RFC 9110 defines HTTP method semantics and Retry-After, which can specify a delay in seconds or an HTTP date. RFC 6585 defines 429 Too Many Requests. An implementation should interpret these signals when present and still keep a local deadline, rather than assuming every provider always supplies the same headers.

A synthetic retry schedule

As an arithmetic example separate from the supplied planner, imagine a policy permitting four attempts within thirty seconds. The first request fails, followed by delays of 1, 2 and 4 seconds before subsequent attempts. Without request durations, total waiting is seven seconds. If each attempt consumes five seconds, total elapsed time reaches twenty-seven seconds.

Now suppose the second response asks for a sixty-second delay. The thirty-second run deadline cannot accommodate it. The correct result is a deferred or incomplete run with a next eligible attempt time, not ignoring the server or sleeping past an undocumented deadline. The numbers are invented to demonstrate budget arithmetic, not recommended settings for a provider.

Coordinate the whole application’s demand

A per-worker limiter can still exceed a provider ceiling when several workers share the same source. Coordinate concurrency and request budgets at the relevant application or network boundary. Do not rotate identities, keys or addresses to evade a limit or access restriction.

The FRED errors page reviewed for this publication states a 120-requests-per-minute threshold, while SEC fair-access documentation states a maximum of ten requests per second. These are dated documentation facts, not target operating rates, reservations or availability guarantees. Recheck current policy before connecting a live integration and use efficient requests well within applicable constraints.

Prevent retry storms and false success

Randomized delay can spread independent clients’ retry times, but it does not justify unbounded repetition. Add cancellation, a maximum elapsed duration and a stable final status. A retry counter should include the first attempt in a clearly documented way so operators can reconcile logs with actual network activity.

For a synthetic test, return two transient errors followed by a valid response and expect exactly three attempts. Then return a permanent schema error and expect no blind retry. Finally exhaust the budget and verify that no complete-data checkpoint is written. These tests inspect the policy’s behavior without sending unnecessary requests to a public service.

Keep recovery evidence without secrets

Record the run identifier, redacted endpoint, status category, attempt count, delay reason and final disposition. Avoid logging full authenticated URLs or response bodies containing confidential material. Error detail should be sufficient to diagnose the failure while remaining appropriate for the audience receiving the report.

The offline samples do not prove live latency or provider uptime. They demonstrate deterministic reactions to simulated failures. A production collector needs managed-cloud execution and durable retry state so an operator’s laptop is never the hidden scheduler or recovery mechanism. A missing data run should remain visibly missing until a later accepted run completes.

The supplied planner is narrower: at most three attempts, retries only for 429 or 503, numeric Retry-After only, and 1/2-second defaults when absent. A requested delay above sixty seconds stops with server_delay_exceeds_local_budget; it is never shortened. No network call, actual wait or HTTP-date parsing occurs.

Keep in the decision record

• Retry only failures that the policy classifies as recoverable.
• Honor server delays without exceeding a declared run budget.
• Coordinate limits across workers and retain an explicit incomplete outcome.

Source notes

1. RFC 9110 — HTTP Semantics, June 2022
2. RFC 6585 — Additional HTTP Status Codes, April 2012
3. FRED API V1 — Errors
4. SEC — Accessing EDGAR Data

Primary specifications support the interface descriptions. All downloadable response fixtures are synthetic. Access and rights checks are dated in the source inventory.

Continue the analysis