Multi-provider failover across LiteLLM, Portkey, OpenRouter and Cloudflare: four decisions the docs make for you

Every gateway people suggest for multi-provider failover — LiteLLM, Portkey, OpenRouter, Cloudflare AI Gateway — supports it. That is the honest answer to “which one supports automatic failover across providers,” which is why the search that brought you here cannot separate them. What differs is the shape of the configuration, and the shape makes four decisions for you:

1. Where the fallback list lives — and therefore who on your team can change it, and whether changing it is a deploy.
2. What counts as a failure — and whether narrowing that list replaces the defaults or adds to them.
3. How long a failed provider stays benched, and whether that timer has anything to do with your retry timer.
4. What happens when the provider you failed over to does not support a parameter in your request. On this one the four ship four different defaults, and one of them will hand your application a 200 for a request it quietly rewrote.

This is the fourth post in a series on where LLM failure domains live. The first was about failures that arrive through the billing layer, the second about how retries amplify them, and the third read seven gateways’ trigger conditions and found the row none of them has. This one is the constructive counterpart: not what they fail to notice, but what you are actually choosing when you turn failover on. Everything below is from the four products’ own documentation as it stood on 2026-08-30, with the config keys quoted verbatim. Where a document does not answer a question, this post says so rather than guessing.

“Which gateway supports it” is the wrong question

First, what these four things actually are, because they are not the same kind of product and the differences later follow from that.

The middle column is the one that matters for everything below: who is running the thing that fails over. When it is your own process, the fallback list is yours to edit and redeploy. When it is somebody’s edge, the list travels in the request.

Now here is each product’s own description of the feature, which is where the confusion starts: they all sound identical.

LiteLLM declares fallbacks as a mapping from a primary model group to an ordered list of backups, and states that “fallbacks are done in-order.” Portkey declares a strategy with "mode": "fallback" and an array of targets. OpenRouter takes provider.order — a list of provider slugs — in the request body, with allow_fallbacks defaulting to true. Cloudflare AI Gateway takes an array of provider objects on its Universal endpoint and, in its own words, “you can add as many fallbacks as you need, just by adding another object in the array.”

Four products, one feature, four sentences that mean roughly the same thing. So the useful question is not which of them has the feature. It is which of the following four decisions you are comfortable letting the default make.

Decision 1 — Is your fallback a list, a strategy, or a request field?

The four put the fallback declaration in different places, and the place determines who on your team can change it and how fast.

LiteLLM’s version is a server-side mapping. It has two more variants alongside the general one — context_window_fallbacks and content_policy_fallbacks — which is a design worth noticing: it treats “the request was too long for this model” and “this model refused on policy grounds” as different problems from “this provider is down,” each with its own backup list. The other three have a single fallback list, so all three failures share one escape route.

Portkey’s targets are the most composable of the four. Per its documentation, “fallback targets are fully composable — each target can be a load balancer, a conditional router, or another fallback,” with no stated nesting limit. That buys expressiveness, and for teams that review config in pull requests it costs reviewability: a nested config is harder to read than a flat list.

OpenRouter and Cloudflare both put the list in the request, which means the application owns it. For OpenRouter that is one field. For Cloudflare it is more than that — each element of the array carries its own provider, endpoint, headers and query, so the per-provider request differences are yours to write out, in full, in every call. That is either honest or laborious depending on how many providers you are chaining.

The practical consequence has nothing to do with elegance. If the list lives in server config, changing failover order is a deploy. If it lives in the request, it is an application change — which can be a feature flag, and can also be forty call sites.

Decision 2 — What counts as a failure, and does specifying it replace the default?

Every one of the four lets you narrow the trigger set. The trap is what your narrowing does to the defaults.

Portkey has two separate on_status_codes fields, on two different objects, and they are worth keeping apart. On the fallback strategy, the default is any non-2xx response, and on_status_codes narrows that; the documentation does not say whether your list replaces the default or adds to it. On retries, it does say. The retry default set is [429, 500, 502, 503, 504, 529], and the docs are explicit that supplying on_status_codes means retries happen “only on those codes—not the defaults.” That is a replace, not a merge:

you write:     on_status_codes: [429]     (on retries)
what you get:  retries on 429 only — 500, 502, 503, 504 and 529 are no longer retried

Both behaviours are defensible; only one of them is what most people mean when they add a status code to a list.

LiteLLM lets you go finer still, keying policy to the exception class rather than the status code — RateLimitErrorRetries, ContentPolicyViolationErrorRetries and their allowed_fails counterparts each take their own number, so “retry rate limits aggressively, give up on content-policy refusals immediately” is expressible without touching the others.

OpenRouter’s default is not a trigger list at all. By default it spreads your traffic across providers, favouring cheaper ones (it weights them by the inverse square of price) and skipping any that have errored in the last 30 seconds; the ones it does not pick are your fallbacks. The important part is what turns that off. Per the docs, specifying sort or order disables load balancing and makes routing deterministic — a fixed list in the order you wrote. So if you set order purely to express a preference, you have also opted out of price-weighted balancing across the whole pool: a side effect that is documented, easy to miss, and invisible in your bill until the month closes.

Cloudflare triggers on an error response or a configured timeout. Every response carries a cf-aig-step header — cf-aig-step: 0 when the first provider served it, 1 when the second did, and so on. One value on every response, no metrics pipeline required.

The previous post in this series read these trigger lists across seven products and found they agree more than the marketing suggests. The point here is the adjacent one: agreeing on the list does not mean agreeing on what happens when you edit it.

Decision 3 — Cooldown and retry are two clocks

Once a provider has failed, two independent timers start: how long before this request tries somewhere else, and how long before that provider is considered healthy again. The second one is the cooldown — a provider that just failed gets benched for N seconds, and the gateway will not send it anything until the bench time is up. The two are configured separately, and the question that decides your worst-case latency is which one runs first: does the gateway exhaust its retries against the failed provider before moving on, or does it move on immediately? One of these products documents the answer. Another does not.

LiteLLM documents it. Its reliability page states that “if a call fails after num_retries, LiteLLM falls back to another model group,” and the configuration example annotates the retry count as “retry call 3 times on each model_name.” So the order is explicit: the retry budget is spent inside a model group first, and only when it is exhausted does the request cross to the next group. Its routing documentation lists the knobs for both clocks — allowed_fails (the failure count that trips a cooldown), cooldown_time (how long that provider entry stays benched — LiteLLM calls each provider-and-model entry a “deployment”, which has nothing to do with shipping code), num_retries and retry_after (the minimum wait before a retry). Most are settable globally in router_settings; allowed_fails is documented as a per-entry setting under model_info. It also documents different retry behaviour by error class: exponential backoff for rate-limit errors, immediate retry for generic ones. Note that the stated defaults are not consistent across the page — allowed_fails: 3 appears in the settings block, while the default cooldown_time is given as 5 seconds in one place and the error message quoted for the cooldown state says 60. Read your own version’s config rather than trusting any single line, including this one.

Portkey specifies the ladder but not the ordering. The ladder is exact: attempts documented up to 5, exponential backoff at 1, 2, 4, 8 and 16 seconds, or use_retry_after_headers: true to honour the provider’s own retry-after (with a cumulative 60-second cap per request). What neither its retry page nor its fallback page states is how the two interleave. The difference is not academic. If all five attempts are spent on target one first, a dead provider costs you up to 31 seconds of backoff before the fallback is even tried; if the fallback happens first, you get a fast answer from a different provider. This post is not going to infer which one from the shape of the ladder. If you are building on Portkey and that ordering sits inside your latency budget, it is a question for their support, not for a blog.

Cloudflare puts the retry clock in the same place as the fallback list: the request itself. Its request-handling headers — cf-aig-max-attempts, cf-aig-retry-delay, cf-aig-backoff — let a caller set the retry count and backoff shape per call, which is consistent with an edge service whose fallback array also travels in the request. OpenRouter documents no per-request retry count at all; its fallback array is the retry mechanism. There is no “try provider A three times” — the second attempt is already provider B.

Why this matters beyond configuration hygiene: retries and cooldowns are the two places where a gateway can turn one provider’s bad minute into your whole fleet’s bad hour. That mechanism was the subject of the second post in this series, and nothing about multi-provider failover makes it smaller — a fallback chain multiplies the number of upstreams your retries can reach.

Decision 4 — The one that rewrites your request

Here is the decision that has the least documentation and the largest blast radius. You have failed over from provider A to provider B. Your request contains a parameter — say response_format for JSON mode, or a tools array — that provider B does not support. What happens?

The four products answer differently, by default:

The bottom two rows are not a third and fourth option so much as an absence of one. Where LiteLLM and OpenRouter each make a decision on your behalf, Portkey and Cloudflare pass the request through as written — which means your failure mode depends on which provider you failed over to, not on the gateway. Two upstreams can answer the same unknown field differently, and nothing in the middle will normalise that for you.

LiteLLM’s documented default is to raise rather than proceed: sending a parameter a model does not support triggers an error. Setting drop_params=True — globally, in litellm_settings, or per call — removes unsupported parameters instead, and additional_drop_params lets you name specific ones. It also ships litellm.get_supported_openai_params("model-name"), which is the only programmatic capability query among the four; you can ask, before you route, what the target actually accepts.

OpenRouter’s documented default is the opposite. In its own words, providers “that don’t support all the LLM parameters specified in your request can still receive the request, but will ignore unknown parameters.” Setting require_parameters: true restricts routing to providers that support everything you sent. There is also a soft preference layer — tools, response_format and verbosity are preferred when choosing between providers — but the documentation is clear that this preference never removes a model from the candidate list if no provider supports the parameter.

Read those two rows together, because they are the same scenario with opposite outcomes. On LiteLLM’s default, a failover to a less capable provider fails loudly — it shows up in your error rate and your alerts fire. On OpenRouter’s default, the same failover succeeds, returns HTTP 200, and the parameter you sent is gone. If that parameter was response_format, your application just received prose where it expected JSON — and every layer in between, including your dashboards, saw a successful request.

This is the constructive half of what the previous post called the row none of the trigger matrices has: a failure that arrives as a well-formed response. That post’s conclusion was that no gateway tracks it. The addition here is narrower and more actionable: for the specific case of parameter compatibility, three of these four products do give you a control — require_parameters, drop_params, an explicit override_params per target — and the control is off, or set to the permissive side, unless you turn it on. Failing over is not the risky part. Failing over to a target with a different capability surface is.

What none of the four configures for you

Three gaps showed up in all four documents, which is worth saying plainly rather than implying by omission.

Token accounting across providers. None of the four documents, in its failover pages, addresses the fact that the provider you fail over to may tokenize differently and bill in different units. Your fallback can be correct and your cost model still wrong.

Context window differences. LiteLLM is the only one of the four that treats this as its own problem, with a separate context_window_fallbacks list. The existence of that separate key is itself the argument: a request that is too long for the backup is not the same incident as a backup that is down, and a single ordered list cannot express both.

Whether the fallback target is a good idea. No configuration language here has a way to say “fall back to this model only for requests that do not need tool calling.” Portkey’s conditional routers come closest, as a general routing mechanism rather than a capability check.

One more thing this post does not have: latency or success-rate measurements. Nothing here was benchmarked. Every number above is a documented default or a documented limit, read on 2026-08-30 and cited to its page. If you want to know which of these adds the least overhead in your region with your traffic shape, that requires a test rig none of these documents can substitute for — and a claim without one would be exactly the kind of number this blog does not print.

A pre-flight checklist before you turn failover on

Six questions, each pointing at one of the decisions above. All six are answerable from your own config in a few minutes.

1. Where does the fallback list live, and what does changing it cost? Server config means a deploy; request body means an application change. Decide which one you want at 3 a.m., before it is 3 a.m.
2. Does your narrowed status-code list replace the defaults or extend them? On Portkey’s retries it replaces. Write out the full set you want rather than the delta you are thinking of.
3. Have you disabled load balancing without meaning to? If you set order or sort on OpenRouter to express a preference, deterministic routing is now on and price-weighted balancing is off.
4. Which parameters in your request does every target support? Start by listing the ones that change the response shape — response_format, tools, structured output — and check those first. On LiteLLM, get_supported_openai_params() answers this in code. On the other three there is no such query: it means reading each provider’s own model page and writing the answer down yourself. That asymmetry is worth knowing before you pick.
5. What is your default when a target does not support one? Raise, drop, or ignore. All three are legitimate; only one of them is on your account right now, and if you did not choose it, the vendor did.
6. Can you tell after the fact which provider served a request? Cloudflare’s cf-aig-step gives it on every response. If your gateway does not, log the served provider yourself — the alternative is finding out at the end of the month, from the bill.

Which constraint puts you where

None of this is a ranking, and the four decisions do not add up to one. But they do narrow the field, and the narrowing runs from your constraints to a class of configuration model — not to a product name.

• If your failover order has to change at 3 a.m. without a deploy → the list has to live in the request or in a config object you can swap, not in server config you ship. That splits the four cleanly, and it is the split with the shortest path from incident to fix.
• If your requests carry response_format, tools, or anything else whose absence changes the response shape → you need the raise-or-exclude side of Decision 4. A default that ignores unknown parameters is usable, but only with validation you write yourself on every path — budget for that work rather than discovering it.
• If nobody on the team wants to review nested configuration in a pull request → composability is a liability, not an asset. A flat ordered list you can read in one glance is worth more than an expressive one nobody checks.
• If you cannot answer “which provider served this request?” from your logs today → start with whichever product hands you that for free, because every other decision here gets easier to evaluate once you can see the outcome.

The choice is a config model, not a product

If you came in asking which gateway supports automatic failover across multiple providers, the answer is all four, and it was never the constraint. What separates them is where the list lives, whether your edits to a trigger set replace or extend it, whether the two clocks are yours to set, and what each one does with a request the new target cannot fully honour.

Those are not tie-breakers to check after you have picked a product. They are the pick. A team whose on-call engineer can change failover order without a deploy needs a different one of these four than a team that wants that order in code review. And on the fourth decision: a gateway that raises on an unsupported parameter puts the decision back on your side of the wire; one that ignores it makes the decision for you, correctly or not. Either can be the right default for your workload — but only if it is the one you picked.

We are building PiRouter on the side that fails loudly — a request that cannot be served as written should say so, not arrive as a 200 with its shape quietly changed. That is a design position, not a feature list, and PiRouter is not in the comparison above: the four products here are documented and shipping today, and they are worth reading in the original before you trust anyone’s summary, including this one.