# Retry storms: 9K to 100K RPS, and four rules that stop them

> GitHub's August 17 outage started with an autoscaling policy and spent its last four and a half hours fighting its own retries — one token service went from 7–9K to 70–100K requests per second on retries. The same arithmetic is sitting in your LLM SDK defaults and in your gateway's failover path. Here is the mechanism, the week's LLM-provider samples of it, and the four rules a gateway owes the providers behind it.

- Published: Aug 21, 2026
- Author: Leo Kaka, Engineering
- Tags: reliability, failover, gateway
- Canonical: https://pirouter.ai/blog/retry-storms

---
GitHub's August 17 outage has an official root cause — a service-mesh proxy ran out of
capacity and the autoscaler was watching the wrong metric — and it is the least interesting
part of the report. The interesting part is what the system did next. Per GitHub's own
[incident report](https://www.githubstatus.com/incidents/zkxwbgr0cnmx), "optimistic retry
logic" overloaded the internal load balancers, and later a latent retry bug in VS Code took
the Copilot Token Service from its normal 7–9K requests per second to 70–100K.

Those are two measurements of the same incident — ~20% errors site-wide, 10x traffic on one
service behind it — and the second one is what kept the incident open for its last four and
a half hours. **Retries without a budget are not resilience. They are an amplifier, and the
amplifier is on by default in your LLM SDK, in your agent loop, and — if you are not
careful — in your gateway.** A retry budget, for the rest of this post, means a cap on
retries expressed as a fraction of recent successful calls — not a fixed count per request.
The difference between those two is the difference between a bad hour and a bad afternoon.

This is the second post in a series on where LLM failure domains actually live. The
[first one](/blog/billing-failure-domain) was about where failures come from. This one is
about how they get bigger.

## What actually happened on August 17

The numbers below are from GitHub's resolved-incident post and the 36 timeline updates on
its status page, and from the [follow-up on the GitHub Blog](https://github.blog/news-insights/company-news/the-august-17-outage-and-the-work-ahead/)
by Vlad Fedorov. Where secondary write-ups disagree with the official record, the official
record wins here. One such disagreement up front: the resolved post dates the incident from
13:28, while the first public status update went out at 13:40 — the table uses the former.

![GitHub's resolved incident report for August 17, 2026](/blog/images/retry-storms-github-rca.png "Fig. 1 — the resolved incident report. Two separate retry mechanisms appear in one paragraph. Source: [GitHub Status](https://www.githubstatus.com/incidents/zkxwbgr0cnmx).")

| UTC | What the record says |
|---|---|
| 13:28 | Incident start. At peak (reported 13:45–14:24): web/API error rates ~20%, archive and raw-content downloads ~50%. SAML/OIDC, SCIM, Team Sync affected |
| 16:36 | "Most services recovered" as the Central US datacenter recovered |
| 16:59 | API, Actions, Git, Issues, Pages, PRs, Webhooks declared mitigated — Copilot not on the list |
| 17:30 onward | Three relapses after "mitigated": Git Operations (17:30–18:23), Issues (17:36–20:22), API Requests (18:48–19:01) |
| 18:03 | Actions recovers |
| 19:13 | "We have partially disabled authentication token retries and have seen improvement" |
| 21:02 | Copilot Token Service fully recovered |
| 21:15 | Resolved. 7h 47m |

Read the mechanism in order. The trigger was capacity: a new traffic peak, an Istio sidecar
(the proxy process that sits next to each service and carries its traffic) that could not
scale because its policy "watched host service but not sidecar limits," and then four
HAProxy nodes — the front-door load balancers — exhausting their flow limits, roughly the
cap on connections each can carry at once. That degraded the path every request takes to get
authenticated. That is a bad hour. What turned it into a bad afternoon is stated
plainly: "The problem was worsened by optimistic retry logic which overloaded internal load
balancers." The fix for that stage was not adding capacity. It was *pausing* HAProxy on the
affected nodes, which "produced immediate broad recovery" — less work, not more.

Then it happened again, one layer out. Traffic moved from Central US to Northern Virginia
and was served successfully there. Then, per the report, "delayed replies to a single
internal endpoint triggered a latent retry bug in VS Code that amplified traffic by
approximately 10x." Note the word *delayed*.
The endpoint was slow, not down. A slow reply was enough to start the loop, and once it
started, "a failed token operation could generate many extra requests and enter a retry
loop." The recovery playbook, quoting the report: temporarily reduce gateway retry logic,
block inbound token requests at the load balancer with a 403, and "gradually ramp back up
traffic per-site to allow callers to succeed."

Every mitigation that worked in the recovery phase was a version of *send less* — pause a
node, block a loop, ramp back gradually — and the ones aimed at the token service were
specifically *stop retrying*. The blog post's
first listed follow-up item says the same thing in policy form: "consistent retry limits,
retry budgets, and variable timeouts across service-to-service interactions to prevent retry
storms and cascading load."

## It happened eleven days earlier, too

The August 6 Actions incident has a different trigger — a routine deployment that exposed a
capacity weakness — and the same second act. From that [incident's
report](https://www.githubstatus.com/incidents/qcvjkzcs7j74): "runners were getting assigned
jobs that were no longer valid and then getting stuck retrying those jobs, preventing them
from picking up valid work." The mitigation was "to prevent runners from repeatedly
attempting to acquire invalid jobs." Only then did the queues drain.

Two incidents in one month, two unrelated subsystems, two different retry mechanisms, same
family — August 6 was head-of-line blocking rather than traffic amplification, but the
cure was the same verb. StatusCake's [write-up](https://www.statuscake.com/blog/what-broke-github-on-august-17-and-how-retries-made-the-incident-worse/)
put it in one line about August 17 alone — the internal optimistic retries and the VS Code
loop: "Two unrelated systems, two separate retry mechanisms, the same failure pattern, on
the same day." Add August 6 and it is three mechanisms in eleven days. It is a pattern, and
patterns do not live in components.

## The amplification, step by step

![Sequence diagram: one failed call becomes nine as client retries multiply gateway retries against a slow upstream](/blog/images/retry-storms-amplification.png "Fig. 2 — the multiplication nobody configured on purpose. Each layer's retry count is reasonable on its own; the upstream sees the product.")

Walk through it with defaults that ship in real code. Your SDK tries 3 times — one call plus
2 retries. Your gateway, trying to be helpful, tries 3 times per SDK attempt. **The upstream
is slow, not down** — it is not returning errors, just taking long enough to trip your
timeouts, and a timeout looks exactly like a failure to every retry loop above it. One user
request becomes 3 × 3 = 9 upstream requests before anyone has seen an error message. The
user sees a spinner, hits refresh, and it is 18. Multiply by every user who is awake. GitHub
measured one such loop at roughly 10x, and nobody wrote `retries: 10` anywhere.

Lorin Hochstein's [post on the
incident](https://surfingcomplexity.blog/2026/08/19/github-autoscaling-and-the-component-substitution-fallacy/)
names the trap that makes this hard to fix organizationally, borrowing David Woods's term:
the *component substitution fallacy*, the belief that reliability comes from finding and
replacing the defective part. His argument is that your system is full of bugs that have
not fired yet and is not down, so defects alone are not what takes you down — interactions
are. The sidecar policy was a
defect. The retry storm was an interaction between the policy, the traffic shape, the
HAProxy flow limits and every client's idea of what to do on a timeout. You can fix the
policy tomorrow. The interaction is still there, waiting for the next slow endpoint.

## Now replace "Copilot Token Service" with your LLM provider

The week's LLM-provider forums supplied the same mechanism in a different protocol. Two
threads, both already familiar to readers of the previous post, but read this time for what
the *clients* did. Two terms before we start: a `429` is the HTTP status for "too many
requests — you are being rate-limited," and `Retry-After` is the optional header a server
can attach to it saying how long to wait. Whether that header is present, and whether
anyone reads it, is most of this section.

On the Google AI developer forum, the [tier-downgrade
thread](https://discuss.ai.google.dev/t/urgent-billing-account-downgraded-from-tier-3-to-tier-1-unexpectedly-service-paused/142555)
picked up new reports on August 19. One user wrote that the downgrade "always resolves itself
within a few hours, when the system remembers we're actually on Tier 3." Another, on pre-pay
with a €1,300 balance: "Now lasting for 8h… earlier downgrades were only 1h… Production down
- backup in place." That last clause is the only correct client behavior in the thread. A
provider that is quietly capping you at Tier 1 and pausing you for eight hours is not an
endpoint to retry into. It is an endpoint to route around.

![NVIDIA developer forum thread on persistent 429 errors](/blog/images/retry-storms-nim-429-thread.png "Fig. 3 — the 429 thread. The rate-limit semantics were undocumented, so clients guessed. Source: [NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/api-error-429-help/376113).")

On the NVIDIA NIM forum, the [429
thread](https://forums.developer.nvidia.com/t/api-error-429-help/376113) is a catalog of
clients flying blind. One user reported making requests "with 5 to 10-minute intervals
between each one" and still getting 429s; another let it "sleep five minutes between every
request" with the same result; a third guessed the limit "resets after 1 hour approx" and
that you get "4-6 request" before it trips again, "even if you implement sleep between API
calls"; and one suspected the provider had clamped down "to stop those users who use AI
agents that are flooding the API with endless requests without sleeping between them." NVIDIA's own FAQ, quoted in the thread, says the
catalog is "a trial experience" and that "the rate of requests will vary per model queried
and may vary based on the number of concurrent users." Translation: in effect, the limit is a
function of everyone else's retries. Nobody in the 41-post thread mentions a `Retry-After`
worth honoring, and the FAQ publishes no number, so every client in it is guessing. Whether
or not the agent-flood theory is right, the structure is GitHub's structure: a constrained
upstream, clients that treat 429 as "try again," no documented recovery window, and a limit
that tightens as the retries arrive.

Line them up:

| | GitHub, Aug 17 | LLM providers, same week |
|---|---|---|
| Upstream state | Slow, then partially failing | Reported quota downgrades; per-model 429s |
| Client reaction | Client-level retry loop (VS Code) | SDK defaults + agent loops + humans re-running |
| What the client got wrong | Slow ≠ failed: a delayed reply was treated as a failure and retried | A 429 with no documented limit, and no `Retry-After` anyone reports, was treated as "try again soon" |
| What stopped it | Reduce retries, 403 the loop, ramp per-site | Routing to a backup (one user); for everyone else, the provider fixing it |

The difference is that GitHub could see the storm from the inside and shut it off. When you
are the client of an LLM provider you cannot see their load balancer. You can only decide
what your side sends. That decision usually lives in a gateway.

## What a gateway owes its upstreams

A gateway that sits between many callers and a handful of providers is the single worst
place to have naive retry logic, because it multiplies whatever the callers already do. It
is also the only place where a budget can be enforced across callers. Four rules, each with
its cost written next to it, because every one of them makes some individual request fail
sooner than it strictly had to.

Three words the table leans on. *Backoff* is waiting longer between each successive retry,
usually doubling. *Jitter* is adding randomness to that wait so that a thousand clients who
failed at the same instant do not all come back at the same instant. A *circuit breaker* is a
switch in front of a provider that trips open when the error rate crosses a line, sends
nothing for a while, then lets a few probe requests through (*half-open*) before trusting it
again; a breaker that keeps tripping and resetting is *flapping*.

![Checklist card: a budget not a count; jitter or nothing; Retry-After outranks your backoff; break per provider](/blog/images/retry-storms-gateway-rules.png "Fig. 4 — the four rules. None of them is novel. All of them were missing somewhere on August 17.")

| Practice | Verdict | Why |
|---|---|---|
| Fixed-interval retry (`sleep 1; retry`) | ✗ | Synchronizes every client into one wave; the upstream sees a heartbeat of spikes |
| Exponential backoff, no jitter | ⚠ | Better than fixed intervals, still synchronized — everyone's second retry lands at the same second |
| Exponential backoff with decorrelated jitter | ✓ | Spreads the retries; cost is higher tail latency for the unlucky request |
| Per-call retry count (`maxRetries: 3`) | ⚠ | Fine in isolation; multiplies across layers. Cap it, but it is not the control you need |
| Per-provider retry budget (share of successful calls) | ✓ | The storm cannot form — once the budget is spent, failures fail fast. Cost: during a real incident, more requests error immediately instead of eventually succeeding |
| Ignoring `Retry-After` | ✗ | The upstream just told you its recovery window; your backoff formula does not know it |
| Retrying on timeout | ⚠ | A timeout is the signal that started GitHub's loop, so it must count against the budget like any failure. Cost: the upstream may still be working on the first attempt, so retry only with a shorter per-attempt timeout and a deadline that the whole chain respects |
| Circuit breaker per provider × model | ✓ | Opens on error rate, probes half-open, closes only on success. Keyed per model as well as per provider, because LLM 429s are usually per model. Cost: a flapping breaker can shed a provider that was about to recover |
| Retrying a half-streamed response on another provider | ⚠ | Safe only if the caller can discard the partial stream; otherwise you bill twice and answer twice. Rule of thumb: once the first token has been forwarded, the request is no longer retryable anywhere — fail it to the caller and let the caller decide |

Here is the smallest implementation of the two that matter most — a budget and a backoff
that defers to the upstream:

```ts
// Per-provider retry budget + decorrelated jitter + Retry-After precedence.
// Budget model (the Finagle/Envoy shape): within a sliding window, retries may
// consume at most `ratio` of the successful calls seen in that window, with a
// small floor so a cold or low-traffic provider still gets *some* retries.
// Lifetime counters would be wrong here: a week of uptime would bank enough
// budget to fund the exact storm this is meant to prevent.
type Budget = {
  windowMs: number; windowStart: number;   // rollover resets the two counters
  successes: number; retries: number;
  ratio: number; minRetries: number;       // e.g. 0.1 and 1 per window
};

function canRetry(b: Budget, now = Date.now()): boolean {
  if (now - b.windowStart >= b.windowMs) {  // window rollover
    b.windowStart = now; b.successes = 0; b.retries = 0;
  }
  // 10 successes at ratio 0.1 buys one retry; the floor buys one regardless.
  return b.retries < Math.max(b.minRetries, Math.floor(b.successes * b.ratio));
}

function nextDelayMs(prevMs: number, base = 200, cap = 20_000): number {
  // Decorrelated jitter: random between base and 3x the previous delay.
  const hi = Math.max(base * 3, prevMs * 3);
  return Math.min(cap, base + Math.random() * (hi - base));
}

function retryAfterMs(res: Response, cap = 20_000): number | null {
  // RFC 9110 allows delta-seconds or an HTTP-date; handle both or you will
  // silently fall back to local backoff on exactly the providers that told you
  // when to come back. Cap it: a Retry-After of 3600 should fail fast, not
  // hold a caller's connection open for an hour.
  const raw = res.headers.get("retry-after");
  if (!raw) return null;
  const secs = Number(raw);
  const ms = Number.isFinite(secs) && secs >= 0 ? secs * 1000 : Date.parse(raw) - Date.now();
  return Number.isNaN(ms) ? null : Math.min(cap, Math.max(0, ms));
}

async function callWithBudget(
  provider: string, req: () => Promise<Response>, budgets: Map<string, Budget>,
): Promise<Response> {
  const b = budgets.get(provider)!;        // budgets are pre-populated per provider
  let delay = 200;                          // = base; the first retry is jittered too
  for (;;) {
    let res: Response | null = null;
    try {
      res = await req();                    // req() owns its per-attempt timeout (AbortSignal.timeout)
    } catch (err) {
      if ((err as Error).name !== "TimeoutError") throw err;
      // A timeout is a failure for budget purposes — this is the case that started
      // GitHub's loop. It is NOT a signal the upstream is idle; it may still be working.
    }
    if (res?.ok) { b.successes++; return res; }
    const retryable = res === null || res.status === 429 || res.status === 503 || res.status === 502;
    if (!retryable || !canRetry(b)) {
      // Fail fast: no budget, no retry. Surface it as a 503 *with* a Retry-After so the
      // caller's own SDK does not re-run this whole loop against us.
      return res ?? new Response(null, { status: 503, headers: { "retry-after": "2" } });
    }
    b.retries++;
    delay = (res && retryAfterMs(res)) ?? nextDelayMs(delay);
    await new Promise(r => setTimeout(r, delay));
  }
}
```

The budget ratio is the whole argument in one number. At `0.1`, a provider that is serving
you normally gets a retry for every ten successes, which absorbs transient blips. A provider
that is failing most requests earns almost no retries — only the floor — so your traffic to
it *drops* as it degrades instead of rising. The floor exists for the provider you just
failed over to: it has zero recent successes with you, and without `minRetries` it would
never earn its first retry. That is the opposite of what VS Code did on August 17, and it is
the property you want from anything that sits in front of a recovering system.

The breaker sits one level up and is mostly configuration:

```yaml
# Per-provider circuit breaker — illustrative shape, not a product config.
providers:
  provider-a:
    breaker:
      key: [provider, model]   # LLM 429s are usually per model, so break per model
      window: 30s              # error rate measured over this window
      open_on_error_rate: 0.5
      min_requests: 20         # don't trip on three failed calls at 3 a.m.
      open_for: 15s            # then move to half-open
      half_open_probes: 3      # let three real requests through; close on success
    retry_budget:
      window: 10s
      ratio: 0.1
      min_retries_per_window: 1
      honor_retry_after: true
      retry_on_timeout: true   # counts against the budget like any failure
    on_open: route_to_fallback        # not: queue and retry later
    on_budget_exhausted: 503 + Retry-After   # tell callers when to come back; don't let them guess
```

Two notes on cost. `min_requests` exists because a breaker that trips on three failures will
shed a healthy provider every time a single user sends a malformed request; set it from your
real per-provider volume, not from a blog post. And `on_open: route_to_fallback` is only a
good idea if the fallback is not also the provider everyone else is failing over to at the
same moment — the NIM thread's "guess what model will be flooded now?" is that failure mode
observed from the receiving end. And a fallback has a price tag of its own; the
[previous post](/blog/billing-failure-domain) made the point that the backup model can bill
several times the primary without anyone noticing until the invoice. Put a budget on the
fallback too.

The shape above is the shape our own gateway is designed around: budgets accounted per
provider rather than per call, an upstream `Retry-After` ahead of any local schedule,
breakers that probe before they trust, and a `Retry-After` of its own on the way out when
the budget is gone. Whether a design survives contact with a real incident is a question
only an incident can answer, which is why this post is about GitHub's and not ours.

## Go count your multipliers

Three things to do before the next provider has a slow afternoon.

Open your LLM SDK's source and find the default retry count. The common default is 2, with
exponential backoff and no budget. Then find every layer between that SDK and the provider
— your agent framework's tool-call retry, your queue's redelivery, your gateway — and
multiply. The worst case per user action is `∏(retries_i + 1)` across layers: two layers at
2 retries each is 9 attempts — the Fig. 2 scenario — and three layers is 27. If that
multiplied total is above single digits, you have built the VS Code bug, and you will find
out which endpoint triggers it at the worst possible time.

Check what your code does with a 429 that carries no `Retry-After`. If the answer is "retry
on the normal schedule," your backoff is guessing at a number the provider chose not to tell
you, and the NIM thread shows how that guessing game ends.

Decide which layer owns the retry. One of them. The others pass failures through — and the
layer that owns it owes the layers below it the same courtesy it wants from the provider:
when its budget is gone or its breaker is open, it answers with a 429 or 503 *and a
`Retry-After`*, so the caller's SDK backs off instead of re-running the whole chain against
you. A gateway that fails fast but fails silently has just moved the storm one hop closer
to home. The [previous post](/blog/billing-failure-domain) argued that a provider's billing system is part
of your failure domain; this one adds that your own retry policy is part of *theirs*. GitHub
spent the last two hours of August 17 turning its own retries off. The cheaper version is
to not turn them on in four places to begin with.

---

## Sources

### githubstatus.com

- [incident report](https://www.githubstatus.com/incidents/zkxwbgr0cnmx)
- [incident's report](https://www.githubstatus.com/incidents/qcvjkzcs7j74)

### github.blog

- [follow-up on the GitHub Blog](https://github.blog/news-insights/company-news/the-august-17-outage-and-the-work-ahead/)

### statuscake.com

- [write-up](https://www.statuscake.com/blog/what-broke-github-on-august-17-and-how-retries-made-the-incident-worse/)

### surfingcomplexity.blog

- [post on the incident](https://surfingcomplexity.blog/2026/08/19/github-autoscaling-and-the-component-substitution-fallacy/)

### discuss.ai.google.dev

- [tier-downgrade thread](https://discuss.ai.google.dev/t/urgent-billing-account-downgraded-from-tier-3-to-tier-1-unexpectedly-service-paused/142555)

### forums.developer.nvidia.com

- [NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/api-error-429-help/376113)