# Multi-provider failover across LiteLLM, Portkey, OpenRouter and Cloudflare: four decisions the docs make for you

> All four support automatic failover across providers, so the question is not who supports it. Their config models diverge on four decisions — and one of them silently rewrites your request and returns a 200.

- Published: Aug 30, 2026
- Author: Leo Kaka, Engineering
- Tags: reliability, failover, gateway
- Canonical: https://pirouter.ai/blog/multi-provider-failover-setup

---
Every gateway people suggest for multi-provider failover — LiteLLM, Portkey, OpenRouter,
Cloudflare AI Gateway — supports it. That is the honest answer to "which one supports
automatic failover across providers," which is why the search that brought you here cannot
separate them. What differs is the shape of the configuration, and the shape makes
four decisions for you:

1. **Where the fallback list lives** — and therefore who on your team can change it, and
   whether changing it is a deploy.
2. **What counts as a failure** — and whether narrowing that list replaces the defaults or
   adds to them.
3. **How long a failed provider stays benched**, and whether that timer has anything to do
   with your retry timer.
4. **What happens when the provider you failed over to does not support a parameter in your
   request.** On this one the four ship four different defaults, and one of them will hand
   your application a 200 for a request it quietly rewrote.

This is the fourth post in a series on where LLM failure domains live. The
[first](/blog/billing-failure-domain) was about failures that arrive through the billing
layer, the [second](/blog/retry-storms) about how retries amplify them, and the
[third](/blog/failover-signals) read seven gateways' trigger conditions and found the row
none of them has. This one is the constructive counterpart: not what they fail to notice,
but what you are actually choosing when you turn failover on. Everything below is from the
four products' own documentation as it stood on 2026-08-30, with the config keys quoted
verbatim. Where a document does not answer a question, this post says so rather than
guessing.

## "Which gateway supports it" is the wrong question

First, what these four things actually are, because they are not the same kind of product
and the differences later follow from that.

| Product | What it is | How you reach it |
|---|---|---|
| LiteLLM | An open-source proxy you run yourself | Start the server, point your SDK at it |
| Portkey | A hosted gateway (self-hosted option available) | Swap your base URL, attach a config |
| OpenRouter | A hosted service with its own API and billing | Sign up, use their base URL and key |
| Cloudflare AI Gateway | An edge service in front of provider keys you hold | Send to their Universal endpoint URL |

The middle column is the one that matters for everything below: **who is running the thing
that fails over**. When it is your own process, the fallback list is yours to edit and
redeploy. When it is somebody's edge, the list travels in the request.

Now here is each product's own description of the feature, which is where the confusion
starts: they all sound identical.

LiteLLM declares fallbacks as a mapping from a primary model group to an ordered list of
backups, and states that "fallbacks are done in-order." Portkey declares a `strategy` with
`"mode": "fallback"` and an array of `targets`. OpenRouter takes `provider.order` — a list
of provider slugs — in the request body, with `allow_fallbacks` defaulting to `true`.
Cloudflare AI Gateway takes an array of provider objects on its Universal endpoint and, in
its own words, "you can add as many fallbacks as you need, just by adding another object in
the array."

Four products, one feature, four sentences that mean roughly the same thing. So the useful
question is not which of them has the feature. It is which of the following four decisions
you are comfortable letting the default make.

## Decision 1 — Is your fallback a list, a strategy, or a request field?

The four put the fallback declaration in different places, and the place determines who on
your team can change it and how fast.

| Product | Where the fallback list lives | What it looks like |
|---|---|---|
| LiteLLM | Server config (`litellm_settings`) | `fallbacks: [{"primary-model": ["fallback-model"]}]` |
| Portkey | A config object (referenced by the request) | `{"strategy": {"mode": "fallback"}, "targets": [...]}` |
| OpenRouter | The request body, per call | `provider: {"order": ["provider-a", "provider-b"]}` |
| Cloudflare AI Gateway | The request body, per call | An array of provider objects on the Universal endpoint |

[LiteLLM's](https://docs.litellm.ai/docs/proxy/reliability) version is a server-side mapping. It has two more variants alongside the general
one — `context_window_fallbacks` and `content_policy_fallbacks` — which is a design worth
noticing: it treats "the request was too long for this model" and "this model refused on
policy grounds" as different problems from "this provider is down," each with its own
backup list. The other three have a single fallback list, so all three failures share one
escape route.

Portkey's targets are the most composable of the four. Per its documentation, "fallback
targets are fully composable — each target can be a load balancer, a conditional router, or
another fallback," with no stated nesting limit. That buys expressiveness, and for teams that review config in pull requests it costs
reviewability: a nested config is harder to read than a flat list.

OpenRouter and Cloudflare both put the list in the request, which means the application
owns it. For OpenRouter that is one field. For Cloudflare it is more than that — each
element of the array carries its own `provider`, `endpoint`, `headers` and `query`, so the
per-provider request differences are yours to write out, in full, in every call. That is
either honest or laborious depending on how many providers you are chaining.

The practical consequence has nothing to do with elegance. If the list lives in server
config, changing failover order is a deploy. If it lives in the request, it is an
application change — which can be a feature flag, and can also be forty call sites.

## Decision 2 — What counts as a failure, and does specifying it replace the default?

Every one of the four lets you narrow the trigger set. The trap is what your narrowing does
to the defaults.

[Portkey](https://portkey.ai/docs/product/ai-gateway/fallbacks) has two separate
`on_status_codes` fields, on two different objects, and they are worth keeping apart. On the
**fallback** strategy, the default is any non-2xx response, and `on_status_codes` narrows
that; the documentation does not say whether your list replaces the default or adds to it.
On **retries**, it does say. The retry default set is `[429, 500, 502, 503, 504, 529]`, and
the docs are explicit that supplying `on_status_codes` means retries happen "**only** on
those codes—not the defaults." That is a replace, not a merge:

```
you write:     on_status_codes: [429]     (on retries)
what you get:  retries on 429 only — 500, 502, 503, 504 and 529 are no longer retried
```

Both behaviours are defensible; only one of them is what most people mean when they add a
status code to a list.

LiteLLM lets you go finer still, keying policy to the exception class rather than the status
code — `RateLimitErrorRetries`, `ContentPolicyViolationErrorRetries` and their
`allowed_fails` counterparts each take their own number, so "retry rate limits aggressively,
give up on content-policy refusals immediately" is expressible without touching the others.

[OpenRouter's](https://openrouter.ai/docs/features/provider-routing) default is not a trigger
list at all. By default it spreads your traffic across providers, favouring cheaper ones (it
weights them by the inverse square of price) and skipping any that have errored in the last
30 seconds; the ones it does not pick are your fallbacks. The important part is what turns
that off. Per the docs, specifying `sort` or `order` disables load balancing and makes
routing deterministic — a fixed list in the order you wrote. So if you set `order` purely to
express a preference, you have also opted out of price-weighted balancing across the whole
pool: a side effect that is documented, easy to miss, and invisible in your bill until the
month closes.

[Cloudflare](https://developers.cloudflare.com/ai-gateway/configuration/fallbacks/) triggers
on an error response or a configured timeout. Every response carries a `cf-aig-step` header
— `cf-aig-step: 0` when the first provider served it, `1` when the second did, and so on.
One value on every response, no metrics pipeline required.

The [previous post](/blog/failover-signals) in this series read these trigger lists across
seven products and found they agree more than the marketing suggests. The point here is the
adjacent one: agreeing on the list does not mean agreeing on what happens when you edit it.

## Decision 3 — Cooldown and retry are two clocks

Once a provider has failed, two independent timers start: how long before this request tries
somewhere else, and how long before that provider is considered healthy again. The second
one is the cooldown — a provider that just failed gets benched for N seconds, and the
gateway will not send it anything until the bench time is up. The two are configured
separately, and the question that decides your worst-case latency is which one runs first:
does the gateway exhaust its retries against the failed provider before moving on, or does
it move on immediately? **One of these products documents the answer. Another does not.**

LiteLLM documents it. Its reliability page states that "if a call fails after
`num_retries`, LiteLLM falls back to another model group," and the configuration example
annotates the retry count as "retry call 3 times on each model_name." So the order is
explicit: the retry budget is spent inside a model group first, and only when it is
exhausted does the request cross to the next group. Its
[routing documentation](https://docs.litellm.ai/docs/routing) lists the knobs for both
clocks — `allowed_fails` (the failure count that trips a cooldown), `cooldown_time` (how
long that provider entry stays benched — LiteLLM calls each provider-and-model entry a
"deployment", which has nothing to do with shipping code), `num_retries` and `retry_after`
(the minimum wait before a retry). Most are settable globally in `router_settings`;
`allowed_fails` is documented as a per-entry setting under `model_info`. It also documents
different retry behaviour by error class: exponential backoff for rate-limit errors,
immediate retry for generic ones. Note that the stated defaults are not consistent across
the page — `allowed_fails: 3` appears in the settings block, while the default
`cooldown_time` is given as 5 seconds in one place and the error message quoted for the
cooldown state says 60. Read your own version's config rather than trusting any single line,
including this one.

[Portkey](https://portkey.ai/docs/product/ai-gateway/automatic-retries) specifies the ladder
but not the ordering. The ladder is exact: `attempts` documented up to 5, exponential
backoff at 1, 2, 4, 8 and 16 seconds, or `use_retry_after_headers: true` to honour the
provider's own `retry-after` (with a cumulative 60-second cap per request). What neither its
retry page nor its fallback page states is how the two interleave. The difference is not
academic. If all five attempts are spent on target one first, a dead provider costs you up
to 31 seconds of backoff before the fallback is even tried; if the fallback happens first,
you get a fast answer from a different provider. This post is not going to infer which one
from the shape of the ladder. If you are building on Portkey and that ordering sits inside
your latency budget, it is a question for their support, not for a blog.

Cloudflare puts the retry clock in the same place as the fallback list: the request itself.
Its request-handling headers — `cf-aig-max-attempts`, `cf-aig-retry-delay`, `cf-aig-backoff`
— let a caller set the retry count and backoff shape per call, which is consistent with an
edge service whose fallback array also travels in the request. OpenRouter documents no
per-request retry count at all; its fallback array *is* the retry mechanism. There is no
"try provider A three times" — the second attempt is already provider B.

Why this matters beyond configuration hygiene: retries and cooldowns are the two places
where a gateway can turn one provider's bad minute into your whole fleet's bad hour. That
mechanism was the subject of the [second post](/blog/retry-storms) in this series, and
nothing about multi-provider failover makes it smaller — a fallback chain multiplies the
number of upstreams your retries can reach.

## Decision 4 — The one that rewrites your request

Here is the decision that has the least documentation and the largest blast radius. You have
failed over from provider A to provider B. Your request contains a parameter — say
`response_format` for JSON mode, or a `tools` array — that provider B does not support. What
happens?

The four products answer differently, by default:

| Product | Default behaviour | The knob | What you must check |
|---|---|---|---|
| LiteLLM | **Raises an exception** | `drop_params: true` to strip unsupported params instead | Nothing — you will hear about it |
| OpenRouter | **Ignores it** (no error, no signal) | `require_parameters: true` to exclude such providers | Whether the response still has the shape you asked for |
| Portkey | No gateway-level default — the upstream decides | `override_params` per target | What each upstream does with an unknown field |
| Cloudflare | No gateway-level default — the upstream decides | Each array element carries its own full request | Same, per element you wrote |

The bottom two rows are not a third and fourth option so much as an absence of one. Where
LiteLLM and OpenRouter each make a decision on your behalf, Portkey and Cloudflare pass the
request through as written — which means your failure mode depends on *which provider you
failed over to*, not on the gateway. Two upstreams can answer the same unknown field
differently, and nothing in the middle will normalise that for you.

LiteLLM's documented default is to raise rather than proceed: sending a parameter a model
does not support triggers an error. Setting `drop_params=True` — globally, in
`litellm_settings`, or per call — removes unsupported parameters instead, and
`additional_drop_params` lets you name specific ones. It also ships
`litellm.get_supported_openai_params("model-name")`, which is the only programmatic
capability query among the four; you can ask, before you route, what the target actually
accepts.

OpenRouter's documented default is the opposite. In its own words, providers "that don't
support all the LLM parameters specified in your request can still receive the request, but
will ignore unknown parameters." Setting `require_parameters: true` restricts routing to
providers that support everything you sent. There is also a soft preference layer —
`tools`, `response_format` and `verbosity` are preferred when choosing between providers —
but the documentation is clear that this preference never removes a model from the candidate
list if no provider supports the parameter.

![Four-row list card titled "Same request, four defaults": LiteLLM raises an exception; OpenRouter ignores the parameter and returns HTTP 200 with the response shape changed; Portkey has no gateway-level default so the upstream decides; Cloudflare is the same, per array element](/blog/images/multi-provider-failover-setup-fig.png "The four defaults side by side. Two products decide for you, two hand the decision to whichever upstream you failed over to.")

Read those two rows together, because they are the same scenario with opposite outcomes. On
LiteLLM's default, a failover to a less capable provider fails loudly — it shows up in your
error rate and your alerts fire. On OpenRouter's default, the same failover succeeds, returns HTTP 200, and the
parameter you sent is gone. If that parameter was `response_format`, your application just
received prose where it expected JSON — and every layer in between, including your
dashboards, saw a successful request.

This is the constructive half of what the [previous post](/blog/failover-signals) called the
row none of the trigger matrices has: a failure that arrives as a well-formed response. That
post's conclusion was that no gateway tracks it. The addition here is narrower and more
actionable: for the specific case of parameter compatibility, three of these four products
do give you a control — `require_parameters`, `drop_params`, an explicit `override_params`
per target — and the control is off, or set to the permissive side, unless you turn it on.
Failing over is not the risky part. Failing over to a target with a different capability
surface is.

## What none of the four configures for you

Three gaps showed up in all four documents, which is worth saying plainly rather than
implying by omission.

**Token accounting across providers.** None of the four documents, in its failover pages,
addresses the fact that the provider you fail over to may tokenize differently and bill in
different units. Your fallback can be correct and your cost model still wrong.

**Context window differences.** LiteLLM is the only one of the four that treats this as its
own problem, with a separate `context_window_fallbacks` list. The existence of that separate
key is itself the argument: a request that is too long for the backup is not the same
incident as a backup that is down, and a single ordered list cannot express both.

**Whether the fallback target is a good idea.** No configuration language here has a way to
say "fall back to this model only for requests that do not need tool calling." Portkey's
conditional routers come closest, as a general routing mechanism rather than a capability
check.

One more thing this post does not have: **latency or success-rate measurements**. Nothing
here was benchmarked. Every number above is a documented default or a documented limit, read
on 2026-08-30 and cited to its page. If you want to know which of these adds the least
overhead in your region with your traffic shape, that requires a test rig none of these
documents can substitute for — and a claim without one would be exactly the kind of number
this blog does not print.

## A pre-flight checklist before you turn failover on

Six questions, each pointing at one of the decisions above. All six are answerable from your
own config in a few minutes.

1. **Where does the fallback list live, and what does changing it cost?** Server config
   means a deploy; request body means an application change. Decide which one you want at
   3 a.m., before it is 3 a.m.
2. **Does your narrowed status-code list replace the defaults or extend them?** On Portkey's
   retries it replaces. Write out the full set you want rather than the delta you are
   thinking of.
3. **Have you disabled load balancing without meaning to?** If you set `order` or `sort` on
   OpenRouter to express a preference, deterministic routing is now on and price-weighted
   balancing is off.
4. **Which parameters in your request does every target support?** Start by listing the ones
   that change the response shape — `response_format`, `tools`, structured output — and
   check those first. On LiteLLM, `get_supported_openai_params()` answers this in code. On
   the other three there is no such query: it means reading each provider's own model page
   and writing the answer down yourself. That asymmetry is worth knowing before you pick.
5. **What is your default when a target does not support one?** Raise, drop, or ignore. All
   three are legitimate; only one of them is on your account right now, and if you did not
   choose it, the vendor did.
6. **Can you tell after the fact which provider served a request?** Cloudflare's
   `cf-aig-step` gives it on every response. If your gateway does not, log the served
   provider yourself — the alternative is finding out at the end of the month, from the
   bill.

## Which constraint puts you where

None of this is a ranking, and the four decisions do not add up to one. But they do narrow
the field, and the narrowing runs from your constraints to a *class* of configuration model
— not to a product name.

- **If your failover order has to change at 3 a.m. without a deploy** → the list has to live
  in the request or in a config object you can swap, not in server config you ship. That
  splits the four cleanly, and it is the split with the shortest path from incident to fix.
- **If your requests carry `response_format`, `tools`, or anything else whose absence changes
  the response shape** → you need the raise-or-exclude side of Decision 4. A default that
  ignores unknown parameters is usable, but only with validation you write yourself on every
  path — budget for that work rather than discovering it.
- **If nobody on the team wants to review nested configuration in a pull request** →
  composability is a liability, not an asset. A flat ordered list you can read in one glance
  is worth more than an expressive one nobody checks.
- **If you cannot answer "which provider served this request?" from your logs today** → start
  with whichever product hands you that for free, because every other decision here gets
  easier to evaluate once you can see the outcome.

## The choice is a config model, not a product

If you came in asking which gateway supports automatic failover across multiple providers,
the answer is all four, and it was never the constraint. What separates them is where the
list lives, whether your edits to a trigger set replace or extend it, whether the two clocks
are yours to set, and what each one does with a request the new target cannot fully honour.

Those are not tie-breakers to check after you have picked a product. They are the pick.
A team whose on-call engineer can change failover order without a deploy needs a different
one of these four than a team that wants that order in code review. And on the fourth
decision: a gateway that raises on an unsupported parameter puts the decision back on your
side of the wire; one that ignores it makes the decision for you, correctly or not. Either
can be the right default for your workload — but only if it is the one you picked.

We are building PiRouter on the side that fails loudly — a request that cannot be served as
written should say so, not arrive as a 200 with its shape quietly changed. That is a design
position, not a feature list, and PiRouter is not in the comparison above: the four products
here are documented and shipping today, and they are worth reading in the original before
you trust anyone's summary, including this one.

---

## Sources

### docs.litellm.ai

- [LiteLLM's](https://docs.litellm.ai/docs/proxy/reliability)
- [routing documentation](https://docs.litellm.ai/docs/routing)

### portkey.ai

- [Portkey](https://portkey.ai/docs/product/ai-gateway/fallbacks)
- [Portkey](https://portkey.ai/docs/product/ai-gateway/automatic-retries)

### openrouter.ai

- [OpenRouter's](https://openrouter.ai/docs/features/provider-routing)

### developers.cloudflare.com

- [Cloudflare](https://developers.cloudflare.com/ai-gateway/configuration/fallbacks/)