Automatic failover in LiteLLM, Portkey and OpenRouter: what actually triggers it, and the 429 no gateway tracks

Seven LLM gateways — LiteLLM, Portkey and OpenRouter from the title, plus Cloudflare AI Gateway, Kong AI Proxy Advanced, Helicone and LLMGateway — support automatic failover across providers, and every one of them documents it the same way: a list of conditions that count as a failure, a retry policy, and a cooldown after which the failed provider is put back in rotation. The lists agree more than the marketing suggests — 5xx, timeouts and connection errors everywhere; 429 in most; 404 and 400 rarely, and in LLMGateway not at all, because its policy is “server-side failures only.” What no list has is a row for the failure that returns a well-formed response: a rate-limit error that is actually a paused account, a not-found that is actually a retired model, a 200 from a model you did not pin. This post reads the seven products’ fallback and retry documentation verbatim, puts the trigger conditions side by side, and then works through what falls between the rows. The seven are simply the ones whose documentation I read end to end.

It is the third post in a series on where LLM failure domains actually live. The first was about failures that arrive through the billing layer; the second was about how retries make failures bigger. This one is about what the layer in the middle — the gateway — is and is not configured to notice.

“Automatic failover” is a list, so read the list

Each vendor defines the feature in one sentence, and the sentence is always about errors. LiteLLM: “Fallbacks are how LiteLLM does automatic failover. If a call fails after num_retries, LiteLLM falls back to another model group.” Portkey: “By default, fallback triggers on any non-2xx status code.” OpenRouter, for its models array: “any error can trigger the use of a fallback model, including: Context length validation errors, Moderation flags for filtered models, Rate-limiting, Downtime.” Cloudflare AI Gateway: “Cloudflare can trigger your fallback provider in response to request errors or predetermined request timeouts.” Kong’s AI Proxy Advanced is the most precise, because it inherits nginx’s vocabulary: failover happens on whatever is in config.balancer.failover_criteria, and “Client errors don’t trigger failover.”

Read as a group, the feature has four moving parts in every product: a trigger set (which responses count as failure), a retry policy (how many times, how spaced, whether the provider’s Retry-After is honored), a cooldown or breaker (how long a failed target is kept out of rotation — “breaker” as in circuit breaker, which opens to stop traffic and closes to let it through again), and a re-admission rule (what brings it back). The differences between vendors are real, and they are all inside those four parts. Nothing outside them exists in any of the seven documents, with one partial exception noted under the matrix.

One split matters before the tables: where the policy runs. LiteLLM and Kong are software you run, so the numbers below sit in a file you can grep; OpenRouter and Cloudflare AI Gateway run the policy on their side; Helicone and LLMGateway are open source and come in both forms. On a hosted router, every not documented cell below is a question for the vendor.

The trigger matrix: seven gateways, side by side

Both tables were built from the linked pages as they read on 2026-08-25. Start with the one that matters — which response classes make each product move traffic. Legend: ✓ fails over by default · ⚠ opt-in, or with a documented side effect · — the documentation does not say · ✗ not a trigger, stated or inferred from the vendor’s own scope statement.

One footnote on the last row: ✗ means no product documents a trigger on a 200. The partial exception is Portkey’s guardrails, which check the response content and fail over on a 446 — a content check, not state tracking, and “no action is taken for output guardrails on streaming.”

Three things fall out of this table. First, 5xx and timeouts are a universal language; if that is all you need, every product on the list will do and the decision is about everything else. Second, 429 is where the products split: LiteLLM cools a deployment down the instant it sees one, LLMGateway’s list is server-side failures only and leaves 429 out, and Kong makes you ask for it by name. Both positions are defensible — switching immediately gets this user’s request answered now, while a 429 is also the provider telling you to slow down, and rerouting a burst to the next provider is how you export your congestion to someone else’s rate limit. Third, and this is the row I care about: 4xx is treated as the caller’s problem almost everywhere, which is correct for the malformed request that produced most 4xx you have ever seen and wrong for the two 4xx classes this series has been documenting — the paused-account 429 and the retired-model 404.

For anyone who wants to check the cells, the parameters behind them. Not documented means I could not find the parameter on the vendor’s fallback, retry, breaker or timeout pages — not that the product lacks the behavior.

Every cooldown expiry is a retry on behalf of the whole fleet

Look at the re-admission column again. Every product with a cooldown brings a failed provider back on a timer: LiteLLM after cooldown_time, 5 seconds by default; Kong after fail_timeout, 10 seconds by default — once max_fails is at least 1, since the default 0 disables the breaker and the failed target is never taken out at all; Portkey after cooldown_interval, 30 seconds at minimum. OpenRouter’s 30-second window is a ranking rule, with no removal or re-admission semantics documented. None of the three probes before re-admitting by default — LiteLLM’s opt-in health-check routing is the exception, and it comes up below. The clock runs out and the next real user request is the probe.

For a thirty-second blip that is exactly right. For the eight-hour pause one user reported in the Google tier-downgrade thread the arithmetic is different: 28,800 seconds divided by a five-second cooldown is 5,760 re-admissions, and each one costs at least one real request a failed hop before the deployment — LiteLLM’s word for one provider-plus-model entry — is cooled down again. That is a retry storm moved up one level. The per-request retry has a budget, or at least a count; the fleet-level retry — “put it back and see” — has neither. Its interval is a constant, it does not back off, and it does not read Retry-After, because by the time the timer fires the response that carried the header is gone.

LiteLLM deserves credit for doing this multiplication once. It pins the provider SDK to max_retries: 0 and says why: “it is what stops a deployment num_retries: N from being applied twice and turning one request into (1 + N) ** 2 upstream calls.” That is the right instinct, enforced at one boundary. Nobody enforces it at the cooldown boundary, in any product, because cooldown expiry is not modelled as a retry. It should be.

The row none of them has: a 429 that is not congestion

Every product in the matrix treats 429 as one thing. Providers do not. Anthropic’s rate-limit documentation describes what happens when an organization hits its monthly spend cap: “API usage pauses until 00:00 UTC on the first day of the next month … API requests return HTTP 429”, and then the sentence that matters for every gateway above: “The error type is rate_limit_error, the same as for a rate limit, but the response has no retry-after header. Retrying, including the SDKs’ automatic retries, fails until access resumes.” The only distinguishing mark is a nested field, error.details.error_code: "enforced_spend_limit_reached". OpenAI’s equivalent is the insufficient_quota 429 whose message — “You exceeded your current quota, please check your plan and billing details” — is reproduced in their own cookbook. Same status code as a busy afternoon. A rate limit lasts seconds; a spend cap lasts until the first of next month.

Run the three incident classes from the first post through the matrix and the pattern holds:

The first row is the one worth staring at. Per-request failover works — six of the seven products will get the individual request to a second provider. What none of them keeps, inside the routing policy, is the fact. A 429 that has persisted for eight hours while your request rate sits well below your known limit is not a rate limit, it is a state change in the relationship between you and the provider, and the correct responses — open the billing console, switch to prepaid credit where the provider offers it, move the traffic deliberately, stop paying for the probe — are not things a five-second timer can do. Nobody looks because the gateway itself is healthy: the fallback is serving, error rate is flat, and the only symptom is a bill from the wrong provider at the end of the month. Your monitoring can be taught to page on this — the last section says how — but this post is about why the routing policy does not do it for you.

The 404 row has a sharper edge. Kong’s schema is explicit that “the cases of http_403 and http_404 are never considered unsuccessful attempts” for the purpose of the breaker count, and its default trigger set leaves http_404 out; LLMGateway’s “Retries are triggered by server-side failures only” is a policy statement. They are right for the general case — a 404 is your typo far more often than their retirement — and the cost of being right in general is that a retired model produces one failed real request per cooldown cycle until a human edits a config file.

Why the documentation is shaped this way

The gateway sees HTTP. The account is on the other side of the wire. The two products in the matrix with quota-aware routing — Cloudflare’s Dynamic Routing, with rate and budget limits “per your key, per period”, and LiteLLM’s Budget Routing, with per-provider, per-model and per-tag budgets — both enforce limits against your ledger, which the gateway can read, not the provider’s, which it cannot. LLMGateway’s uptime score is the closest thing in the table to state with a duration in it — a 60-minute rolling window, weighted toward the last few minutes — and it is fed exclusively by 5xx, timeouts and connection failures, so a paused account with a valid key scores as perfectly healthy. LiteLLM’s “>50% failures in the current minute” is a state, too, and it resets on the minute. Nobody is doing anything wrong here. The abstraction is HTTP, and HTTP does not have a status code for “your relationship with this vendor changed at 09:14.”

What a gateway could watch instead

Three signals exist today that a gateway could consume. None is exotic; what is missing is the wiring. Scope, stated once: this is the design we are building toward in our own gateway, not a shipped feature — every claim below reads “a gateway can,” and the links go to provider documentation. It does not touch the 200 row; a model substituted behind a 200 is a content-layer problem none of these three signals sees.

A classification with a duration in it. The first post proposed splitting faults into transient and structural, where a 429 becomes structural once it has persisted past a recovery window. The half the matrix is missing is what happens next: a structural fault should not be re-admitted by a timer. It should be re-admitted by a probe — a single request on a dedicated key, whose success is the only event that closes the state. Cost: you keep a small stream of failing probes running against a paused provider, on a line item you can see, instead of 5,760 failing user requests you cannot.

Quota headers, where the provider sends them. Anthropic returns anthropic-ratelimit-requests-remaining, anthropic-ratelimit-tokens-remaining and their -reset timestamps in its responses; OpenAI returns x-ratelimit-remaining-requests, x-ratelimit-remaining-tokens and x-ratelimit-reset-*. A gateway that records these per key sees the tier downgrade before the first 429: the limit header steps down while your traffic does not. Two caveats, stated next to the recommendation: not every provider sends them — the providers in this series’ incidents mostly did not — and a spend-cap pause may arrive without any of them (Anthropic’s spend-cap 429 carries no retry-after, and the docs do not promise the ratelimit headers on it either). Headers are a leading indicator where they exist, not a replacement for the duration rule.

Synthetic probes that take the real path and do not ignore 429. LiteLLM already ships the mechanism: background_health_checks: true with enable_health_check_routing pings every deployment on an interval and removes failures from the pool proactively — their docs say it plainly, “The cooldown system is reactive.” The same page then offers health_check_ignore_transient_errors: true, under which “429 and 408 never affect routing” and only “hard failures (401, 404, 5xx)” count. For congestion that is the right default. For a paused account it is a switch labelled do not notice this. The design change is small: a probe 429 at a time when your production request rate is a fraction of your known limit is not transient, and the probe should say so.

The illustrative shape:

# Provider state tracking — illustrative shape, not a product config.
providers:
  provider-a:
    triggers:                      # the usual list; nothing new here
      status: [429, 500, 502, 503, 504]
      timeout: true
      connection_error: true
    structural:                    # the row the matrix is missing
      persist_429_for: 15m         # 429 past this window at low utilization = state change
      model_lifecycle_4xx: true    # 404/400 with model_not_found|deprecated|retired codes
      quota_headers:
        track: [x-ratelimit-limit-requests, anthropic-ratelimit-requests-limit]
        alert_on_step_down: true   # the limit fell; your traffic did not
    readmission:
      transient: timer             # cooldown expiry, as today
      structural: probe            # only a successful probe on a dedicated key closes it
    probe:
      path: real_completion        # not /health; the completion path is what pauses
      key: probe-key-a             # its spend is its own line item
      treat_429_as: structural     # do NOT ignore transient errors on the probe path

Two costs, because there are always two. A structural state that only a probe can close keeps a provider out longer than a timer would when the incident really was a thirty-second blip at low utilization. How often the structural case happens I cannot tell you — this series’ evidence is a handful of forum threads, not a distribution — which is why persist_429_for should come from your own 429 history, not from this file. And the probe key spends money against a provider that is, by hypothesis, not serving you: at LiteLLM’s default health_check_interval of 300 seconds, 288 minimal completions a day. Small enough to ignore, large enough to exist. That is the price of knowing.

Three lines to grep, five questions to ask

1. Your cooldown. cooldown_time in LiteLLM; fail_timeout and max_fails in Kong; cooldown_interval in Portkey. Then ask what the documentation does not: what happens on the hundredth expiry? If the answer is “the same thing as the first,” you have a fleet-level retry loop with no budget.
2. Whether 404 is in your trigger set, and decide on purpose. Kong’s default leaves it out and never lets it trip the breaker; Portkey’s default fails over on it; LiteLLM fails over, then sends the next user back in five seconds. Each is defensible. Not knowing which one you have is not.
3. Whether your health checks ignore 429. In LiteLLM that is health_check_ignore_transient_errors; if it is on, a paused account is invisible to the probe by configuration.

On a hosted router the same three become questions for the vendor, plus two: is cooldown state shared across your instances or kept per process, and is there a metric or webhook when a target is removed or re-admitted?

And one thing that needs no gateway change: alert, in whatever you already use for dashboards, on fallback share — “provider B has carried more than X % of traffic for longer than Y minutes” — or on the wall-clock duration of the primary’s 429s. That is the counter this post says nobody owns. In all seven documents the trigger lists are stateless, the cooldowns are timers, and the one duration-shaped concept — a per-minute failure rate — resets before your coffee is cold. The providers’ billing systems are in your failure domain; your gateway’s retry policy is in theirs; and the state that connects the two currently lives in a forum thread, eight hours after the fact. Until a gateway owns that counter, your dashboard can. That is the row to add.