Make retries fail small before they become an outage

A checkout service slows down, and every caller decides the safest response is to call it three more times. The dependency now receives extra work precisely when it has the least capacity to handle it. What began as a latency fault becomes a traffic amplifier.

See how one retry policy multiplies shared load

The test has to show the whole attempt tree, not just the final 503. Record one logical operation ID across the client, gateway, application, and dependency. Then count starts, finishes, timeouts, delays, and overlapping work at each layer.

A retry is a new request. It consumes another connection, another queue slot, more CPU, and possibly another database transaction. That cost can be worthwhile for a short transient failure. It becomes dangerous when the failure comes from overload and each caller adds work faster than the dependency can drain it.

Use precise terms in the test report:

• A logical request is the user's one intended operation.
• An attempt is one execution by a retrying layer.
• An in-flight attempt has started and has not finished or been cancelled at the observed boundary.
• Amplification is downstream attempts divided by logical requests for the same time window and scope.
• A retry burst is a cluster of attempts caused by aligned retry schedules.
• A retry budget limits additional work, not merely the delay between attempts.

The arithmetic can become surprising without any large numbers. If an SDK makes up to three attempts and the application wraps that SDK in another three-attempt loop, one logical call can produce up to nine dependency attempts when every inner call exhausts its policy. Add a gateway that repeats the application request up to three times, and the upper bound becomes 27. Those are derived maxima, not observed measurements.

Count the initial call consistently. In this article, maxAttempts: 3 means one initial attempt plus at most two retries. Some libraries expose "retries" instead, where a value of three permits four total attempts. Misreading that setting is a common source of off-by-one amplification. Verify the actual client behavior rather than assuming the option name.

Timeouts and retries solve different problems. A timeout limits how long a caller waits for one attempt. It does not guarantee the remote server stopped working. If the caller starts a replacement while the original continues, one logical request has concurrent in-flight attempts. That overlap is often more damaging than three well-separated attempts.

Backoff spaces later attempts. Exponential backoff increases the possible wait after each failure. A cap prevents the delay from growing without bound. Jitter randomizes the selected delay so many clients do not line up on the same schedule. AWS's Builders' Library describes why retries add load during failure and why backoff and jitter are used to reduce correlated bursts. The policy still needs a maximum attempt count and a caller deadline.

HTTP semantics determine whether repetition is safe. RFC 9110 defines safe methods, idempotent methods, and constraints on automatically retrying non-idempotent requests. A timeout does not prove a write was not applied. Tests for POST-like operations need an application idempotency contract or reconciliation path before enabling automatic retry.

Response codes alone do not define one universal retry policy. A 400 caused by a malformed body will not become valid when sent unchanged. A 503 can be temporary, but another attempt can still be harmful if the server is overloaded. A 429 can carry Retry-After. Your client contract must list eligible methods, status codes, transport errors, and exception cases.

Build a bounded harness before adding concurrency

Start with a controllable dependency that can return a sequence such as 503, 503, 200. It should record each attempt's operation ID, attempt number, start time, finish time, outcome, and whether the connection closed. Keep the harness local or isolated. Never point a deliberate retry storm at a production or shared third-party service.

Make randomness injectable. Production should use a suitable random source for jitter, but unit tests need known values. Make sleep injectable too, so policy tests can assert requested delays without waiting. Real-time component tests should be few and tolerant of scheduler noise.

This TypeScript client implements one explicit example contract for idempotent GET requests. It retries network failures, 429, and 503; uses a per-attempt timeout; honors a supported Retry-After value; applies capped full jitter; and stops after maxAttempts. It does not claim to be a universal policy.

import { randomUUID } from "node:crypto";

export type RetryPolicy = {
  maxAttempts: number;
  attemptTimeoutMs: number;
  baseDelayMs: number;
  delayCapMs: number;
};

type RetryHooks = {
  random: () => number;
  sleep: (milliseconds: number) => Promise<void>;
  now: () => number;
};

const defaultHooks: RetryHooks = {
  random: Math.random,
  sleep: (milliseconds) =>
    new Promise((resolve) => setTimeout(resolve, milliseconds)),
  now: Date.now,
};

export function retryAfterMilliseconds(
  value: string | null,
  now: number,
): number | undefined {
  if (value === null) return undefined;

  const trimmed = value.trim();
  if (/^[0-9]+$/.test(trimmed)) {
    const milliseconds = Number(trimmed) * 1000;
    return Number.isSafeInteger(milliseconds)
      ? milliseconds
      : undefined;
  }

  if (/^[0-9+-]/.test(trimmed)) return undefined;

  const date = Date.parse(trimmed);
  if (Number.isNaN(date)) return undefined;

  return Math.max(0, date - now);
}

export function fullJitterMilliseconds(
  retryNumber: number,
  policy: RetryPolicy,
  random: () => number,
): number {
  const exponential = policy.baseDelayMs * 2 ** (retryNumber - 1);
  const ceiling = Math.min(policy.delayCapMs, exponential);
  return Math.floor(random() * ceiling);
}

export async function retryingGet(
  url: string,
  policy: RetryPolicy,
  hooks: RetryHooks = defaultHooks,
): Promise<Response> {
  if (policy.maxAttempts < 1) {
    throw new RangeError("maxAttempts must be at least 1");
  }

  let lastError: unknown;
  const operationId = randomUUID();

  for (let attempt = 1; attempt <= policy.maxAttempts; attempt += 1) {
    const controller = new AbortController();
    const timeout = setTimeout(
      () => controller.abort(),
      policy.attemptTimeoutMs,
    );

    try {
      const response = await fetch(url, {
        method: "GET",
        signal: controller.signal,
        headers: { "x-operation-id": operationId },
      });

      const retryable = response.status === 429 || response.status === 503;
      if (!retryable || attempt === policy.maxAttempts) {
        return response;
      }

      await response.arrayBuffer();

      const retryNumber = attempt;
      const jitter = fullJitterMilliseconds(
        retryNumber,
        policy,
        hooks.random,
      );
      const serverDelay = retryAfterMilliseconds(
        response.headers.get("retry-after"),
        hooks.now(),
      );

      await hooks.sleep(Math.max(jitter, serverDelay ?? 0));
    } catch (error) {
      lastError = error;
      if (attempt === policy.maxAttempts) throw error;

      await hooks.sleep(
        fullJitterMilliseconds(attempt, policy, hooks.random),
      );
    } finally {
      clearTimeout(timeout);
    }
  }

  throw lastError;
}

The operation ID is created once before the loop and sent on every attempt. That lets the fake dependency group all work caused by one caller request. In a larger client, inject an attempt function with the signature (operationId, signal) => Promise<Response> so unit tests can record correlation without making network calls.

Reproduce three storm shapes and read their timelines

Worked example one: immediate retries on a fast 503.

Configure the fake dependency to return 503 immediately. Send one logical request. If the policy permits three attempts, the recorder should show exactly three dependency calls under one operation ID, no fourth call, and the configured delays between them.

A fast failure is important because it exposes a loop that retries without waiting. A slow failure can make an immediate policy look spaced out by accident. Compare each attempt's start time with the previous attempt's finish time. The delay belongs between those events.

Test the delay function without wall-clock sleeps:

import assert from "node:assert/strict";
import test from "node:test";

test("full jitter stays below the capped exponential ceiling", () => {
  const policy: RetryPolicy = {
    maxAttempts: 3,
    attemptTimeoutMs: 500,
    baseDelayMs: 100,
    delayCapMs: 250,
  };

  assert.equal(
    fullJitterMilliseconds(1, policy, () => 0.5),
    50,
  );
  assert.equal(
    fullJitterMilliseconds(2, policy, () => 0.5),
    100,
  );
  assert.equal(
    fullJitterMilliseconds(3, policy, () => 0.5),
    125,
  );
});

test("Retry-After supports seconds and HTTP dates", () => {
  assert.equal(retryAfterMilliseconds("3", 0), 3000);
  assert.equal(
    retryAfterMilliseconds(
      "Thu, 01 Jan 1970 00:00:05 GMT",
      2000,
    ),
    3000,
  );
  assert.equal(retryAfterMilliseconds("not-a-date", 0), undefined);
  assert.equal(retryAfterMilliseconds("3.5", 0), undefined);
});

The numbers are deterministic results of the shown functions. They are not production latency observations. A random value of 0.5 chooses half the current ceiling. Your actual distribution tests should verify bounds and use seeded samples rather than demand one exact production delay.

Worked example two: the timeout starts an overlapping attempt.

Configure the dependency to keep working longer than the client's attempt timeout and record when it observes a closed connection. If attempt two starts while attempt one is still active at the server, the operation has overlap.

The client trace might show:

operation=op-17 attempt=1 event=start t_ms=0
operation=op-17 attempt=1 event=client-timeout t_ms=800
operation=op-17 attempt=2 event=start t_ms=850
operation=op-17 attempt=1 event=server-finish t_ms=1400
operation=op-17 attempt=2 event=server-finish t_ms=2250

This is illustrative output, not a measured run. It demonstrates 550 milliseconds where both attempts are active at the observed server. The relevant evidence is the ordering, not those chosen durations.

Cancellation behavior depends on the server, framework, transport, and work already started. Do not assert that aborting fetch automatically rolls back a database operation. Inspect the server-side finish or cancellation event. For a write, also inspect durable side effects and reuse the same idempotency key.

The fix may be a longer attempt timeout, a shorter server budget, propagated cancellation, hedging designed for safe reads, or no retry at that layer. Each has a cost. Longer timeouts hold local resources. Aggressive cancellation can discard work that was close to completion. Hedging intentionally creates overlap and needs a separate capacity budget.

Worked example three: retry policies stack across layers.

Turn off one layer at a time. Run the same failure with only the SDK retry enabled, then only the application loop, then only the gateway behavior, and finally the deployed combination. Preserve attempt counts at every hop.

A useful JSON-lines diagnostic can group the recorded events:

jq -s '
  group_by(.operation_id)
  | map({
      operation_id: .[0].operation_id,
      caller_attempts: map(select(.hop == "caller")) | length,
      gateway_attempts: map(select(.hop == "gateway")) | length,
      dependency_attempts: map(select(.hop == "dependency")) | length,
      first_start_ms: map(.start_ms) | min,
      last_finish_ms: map(.finish_ms) | max
    })
' artifacts/retry-attempts.jsonl

When the application configuration says three attempts but the dependency sees nine, look for a nested retry layer before accusing the counter. SDKs, service meshes, proxies, and job runners can all repeat work. Verify which ones are configured in your system. Do not assert that a named product retries a particular failure unless its current documentation and deployed configuration confirm it.

A near-miss has the same rising latency but no repeated attempts. Queue backlog can make requests finish later while the logical-to-downstream ratio stays 1. Pool starvation can prevent attempts from starting at all. A storm shows extra starts; a backlog shows growing wait time. Capture both count and queue timing.

Separate a retry storm from the fault that triggered it

The triggering fault and the amplification are two defects only when the policy violates its contract. A dependency may legitimately return 503. The caller still owns its maximum attempts, delay, deadline, and retry eligibility.

Start diagnosis with four counts for the same interval: incoming logical requests, caller attempts, dependency attempts, and completed side effects. If dependency attempts rise faster than logical requests, retries or fan-out are present. If both rise together, ordinary load or replay upstream may be the cause.

Then inspect attempt spacing. Back-to-back timestamps after fast failures indicate missing delay. Repeating spikes at the same interval across many operation IDs indicate synchronized schedules or a periodic source. Widely scattered attempts with a high total count indicate an overly generous budget rather than synchronization.

Check in-flight overlap. An attempt count of three can be safe when each finishes before the next begins. The same count can triple concurrent work if timeouts do not stop server processing. Active-request metrics and per-operation timelines tell those cases apart.

Response eligibility is another boundary. If 400 validation responses are retried unchanged, the rule is wrong. If only connection resets are retried, confirm whether the server might have committed before the reset. Status families are not enough for write safety.

Watch the outer test runner. A component assertion may fail after the client already exhausted three attempts, then the runner repeats the whole test twice. The fake dependency sees nine calls, but only three belong to each scenario run. Keep a testRunId outside operationId so evidence can separate runner attempts from application attempts.

Recovery can produce a second wave. When a dependency becomes healthy, waiting callers, scheduled retries, and queued work may arrive together. Test recovery by releasing a controlled fault and recording the arrival distribution. Do not report a generic average over the entire test because a short spike can disappear in it.

Fan-out is another near-miss. One catalog request may legitimately call inventory once for each distinct item, so the downstream count can exceed the logical request count without any retry. Record the target resource, parent operation, and attempt number together. Nine calls for nine different SKUs are fan-out; three calls for the same SKU after the same failure are retries. If the trace has only a shared operation ID, those shapes can look identical. Add a child-operation identifier before changing the retry policy.

Connection reuse can confuse the opposite diagnosis. Several HTTP requests on one connection are still several attempts, while one request that sends multiple protocol frames is not automatically a retry. Count application-level request starts at the receiving service, then use transport evidence only to explain how those requests travelled. Socket counts alone cannot establish amplification.

Circuit state, if your system uses a circuit breaker, needs its own evidence. An open circuit may reduce dependency calls while returning fast failures locally. A half-open state may permit a limited probe. Exact terms and transitions vary by implementation, so assert only your configured contract. A lower dependency count does not prove callers received an acceptable response.

Fix the budget at one layer and roll it out safely

The first fix is often ownership. Choose the layer that has enough context to decide whether the operation is safe and whether the caller still has time to benefit. Disable or reduce retries at other layers where possible. A single retry policy is easier to reason about than three individually reasonable loops.

Give every logical call an overall deadline. Per-attempt timeout plus backoff can otherwise exceed the caller's useful window. Before sleeping or starting another attempt, calculate whether enough budget remains. Return the original or most useful failure when no useful attempt fits.

Cap total attempts. Backoff without a cap can keep weak traffic alive long after the incident changes shape. Cap the delay too, but remember that many clients at the same cap can synchronize. Jitter remains necessary.

Honor Retry-After only where the contract says it applies. The header can contain a delay in seconds or an HTTP date. Clamp negative dates to zero, reject invalid values, and do not wait past the operation deadline. A server hint does not grant the client unlimited time.

Writes need idempotency or reconciliation before automatic retry. Pass the same idempotency key across attempts for one logical operation. Test a lost response after commit, not only clean 503 responses where no write occurred.

Roll out in observe-only mode. Emit the chosen delay, remaining budget, attempt reason, and final outcome without changing the old policy. Compare the new would-be attempt count with the existing count on controlled traffic. Do not log credentials or full request bodies.

Then enable the policy for one dependency and a small traffic segment. Watch success after retry, added attempts, in-flight calls, pool use, deadline exhaustion, and duplicate-effect alarms. Set rollback criteria before deployment. A higher apparent success rate is not a win if it consumes the dependency's recovery capacity.

CI should run policy arithmetic and fake-dependency cases on every change, with a bounded concurrency scenario in an isolated job:

name: retry-policy

on:
  pull_request:
    paths:
      - "src/http/**"
      - "src/retry/**"
      - "tests/retry/**"
  workflow_dispatch:

jobs:
  deterministic-policy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: npm
      - run: npm ci
      - run: npm run test:retry-policy

  bounded-fault:
    needs: deterministic-policy
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
      - run: npm ci
      - run: npm run test:retry-fault
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: retry-attempt-timelines
          path: artifacts/retry/

The script names and paths are placeholders for project-owned commands. The fault job must cap clients, attempts, and duration. It should target only a local fake or isolated dependency owned by the test.

The trade-off is visible. Fewer attempts can reduce recovery from brief faults. Longer jitter can increase user latency. More instrumentation costs storage. An overall deadline can end a request that might have succeeded later. Choose those costs against the operation's user value and the dependency's capacity, then calibrate with controlled tests rather than invented percentages.

Separate a caller retry loop from queue redelivery

Repeated downstream starts under one logical operation can come from a second mechanism that resembles a retry loop in service logs. A queue can make work available again after a worker fails, loses its lease, or does not finish within the delivery window defined by that system. If the first worker continues running, the replacement delivery overlaps it. The downstream timeline then shows the same operation twice, much like a client timeout that starts attempt two while attempt one continues. Tuning HTTP backoff does nothing to the queue's delivery lifecycle.

Separate the causes at the point where work enters the process. A caller retry produces another client attempt before a new queue message or synchronous dependency call. Queue redelivery produces another delivery of the same business message without another caller-side attempt. Preserve the stable message identity, the delivery-attempt evidence the queue actually exposes, worker start and finish times, and the acknowledgment or completion outcome. Do not invent a generic redelivery field when the queue does not provide one. In that case, use the stable message identity plus receive and completion records owned by the consumer.

The grouped diagnostic from the storm harness should be read as a funnel. caller_attempts counts starts made by the policy under test. gateway_attempts shows whether an intermediary multiplied those starts. dependency_attempts shows the final pressure on the target. first_start_ms and last_finish_ms bound how long the operation occupied the system; they do not identify the repeating layer by themselves. A healthy single-attempt path has one start at every applicable hop. A healthy retry path may have more caller and dependency attempts, but they stay within the declared budget and do not overlap when overlap is forbidden.

For queue work, a broken case can show one caller attempt, one produced message, and two worker starts for the same message identity. That evidence clears the HTTP retry loop even though the dependency sees two calls. A caller retry bug instead increments caller starts and usually creates separate downstream attempt records before the queue consumer is involved. The misleading value is a high dependency-to-caller ratio. Fan-out, redelivery, and nested retries can all raise it. Ownership comes from the hop where the count first increases, not from the largest count at the end.

Spacing can also mislead. Redeliveries may recur at a regular interval that resembles fixed backoff. Compare the second worker start with the first delivery's completion or lease lifecycle, and compare caller logs for a recorded retry decision. A backoff defect has another caller decision and requested delay. A delivery defect has no such decision, while the same message becomes eligible for another worker. When neither boundary is recorded, the timestamps support a hypothesis but do not prove it.

An established HTTP retry suite usually assumes every repeated target call came from the client under test. Land hop labels and stable business-message correlation before adding queue-backed scenarios to that suite. Next, split the oracle: one assertion bounds client attempts, another bounds deliveries per message, and a third counts target starts. Keep the old aggregate threshold in report-only mode until historical cases have been classified. Otherwise a newly visible redelivery can look like a regression caused by the instrumentation itself.

Roll out policy changes one repeating layer at a time. First prove the deterministic client budget with the queue removed from the path. Then hold that client policy fixed while exercising one controlled redelivery. If the queue supports extending work ownership, test both successful extension and worker loss under the configuration your system actually uses. Finally, run the combined path with bounded concurrency and observe the first multiplication point. The rollout is working when each synthetic fault increases only its intended counter and when disabling client retries does not falsely appear to cure consumer redelivery.

The queue-side fix has a different trade-off from backoff. A longer delivery window can reduce overlapping redelivery for slow legitimate work, but it can also delay reassignment after a worker truly dies. Extending ownership while work progresses can reduce that delay, at the cost of control traffic and more state transitions to test. Shortening the work unit may improve recovery but introduces checkpoints and partial-progress handling. Those costs must be chosen against real task duration and recovery requirements, not copied from the HTTP attempt timeout.

Ownership follows the first duplicate start. The client-library owner supplies retry decisions, delays, and operation IDs. The producing service supplies the stable business-message identity. The queue platform owner supplies delivery and acknowledgment evidence. The consumer owner proves whether the first worker was still active and whether the effect is safe under redelivery. The dependency owner supplies target-side start and finish events. A handoff should contain one ordered timeline with all of those identities, the configured budgets in effect, the first layer where one became many, and the durable side-effect count.

This technique does not catch a resource leak on a request that always succeeds once. A connection, file handle, or worker slot can accumulate without any retry counter increasing. The attempt funnel stays perfectly healthy while capacity decays across hours. That needs a soak or resource-lifecycle test with process-level observations, not a retry-storm assertion.

Do not use a retry when it cannot improve the outcome

Do not retry deterministic validation failures with the same request. Change the input or return the error. Repetition only adds load.

Avoid automatic retry for a non-idempotent write unless the application knows the action is safe to repeat or can determine that the first attempt was not applied. A connection failure before the response is ambiguous from the client's perspective.

Do not add retries to hide a dependency that consistently misses its service objective. Retries can increase apparent success while consuming capacity and stretching latency. Fix the underlying capacity, query, or dependency contract.

Skip a storm test against shared staging when other teams depend on it. Even a "small" fault can combine with their traffic and hidden retries. Use a local fake for policy, an isolated environment for resource behavior, and a carefully approved resilience exercise for broader interactions.

Do not assert exact millisecond sleep timing in ordinary CI. Schedulers and loaded runners introduce noise. Assert the selected delay through an injected sleep function, then keep a small number of real-time checks with ranges.

A circuit breaker is not a mandatory retry feature. It can protect a dependency, but it also introduces state, recovery probes, and fast-failure behavior that clients must handle. Use it only when the architecture needs that extra state and recovery behavior.

Finally, do not call every burst a retry storm. Prove repeated attempts under shared logical IDs. A cron wave, reconnect wave, queue release, autoscaling delay, or ordinary traffic spike needs a different fix.

A good resilience test deliberately makes failure boring. It bounds the blast radius, labels every attempt, and stops on schedule. The cost is lower peak recovery probability for individual calls. The benefit is that one struggling dependency does not recruit every caller into making the incident worse.