API Testing / Scenario

The Retry Storm That Took Everything Down

A slow dependency plus infinite timeouts plus blind retries equals a cascading outage. Diagnose it and design the resilience.

Difficulty
Boss
Format
Scenario
Points
300
Estimate
20 min

// MISSION BRIEF

Your Mission

Service A calls payment service B. B got slow. A had no client timeout and retried every failure immediately, including a non-idempotent charge. Threads exhausted, A fell over, and the retries made B's bad day worse.

This is a boss fight in resilience: find the cause, stop the double-charges, and specify timeouts, backoff, and a circuit breaker that hold under pressure.

// FIRST CONTACT

Battle teaser

First artifact

Incident timeline

The primary cause of A's OUTAGE (beyond B being slow):

  1. AA DNS misconfiguration
  2. BNo client timeout plus immediate retries against a slow dependency exhausted A's threads/connections, a cascading failure amplified by retries
  3. CA did not have enough servers
  4. DB being slow is the only cause; A did nothing wrong
Answers, scoring, hints, and the full battle stay sealed.

// SKILL TAGS

apitimeoutsretriesresiliencecircuit-breaker