# How to measure the latency an LLM gateway adds: six questions to ask before you believe a number

> Gateway latency numbers disagree because they measure different things. Five axes decide what any of them means: stubbed or real upstream, overhead-as-subtraction or end-to-end, P50 or P99, the load shape, and ratio versus duration. Worked through one public load test whose harness is readable — maximhq's own CI run of Bifrost — where the published overhead table reports a minimum of 999.93ms and a mean above its own P90, because the calibration is subtracted percentile by percentile. Includes the ten-second coherence check that catches it.

- Published: Sep 5, 2026
- Author: Leo Kaka, Engineering
- Tags: gateway, reliability, agents
- Canonical: https://pirouter.ai/blog/how-to-measure-gateway-latency

---
You cannot compare two LLM gateway latency numbers unless you know five things about
each: whether the upstream was stubbed or real, whether the number is an overhead
subtraction or an end-to-end measurement, which percentile it is, what load shape
produced it, and whether it is framed as a ratio or a duration. Change any one of those
and the same gateway on the same hardware will report a figure that differs by orders of
magnitude — honestly, with nobody lying. That is why the benchmarks you find disagree.
And there is a sixth thing, which no publisher has to tell you because you can check it
yourself: whether the numbers in front of you are even mutually consistent. Below, one
published table fails that check.

This post is not a ranking. We build a gateway, which means we know exactly how these
numbers are produced and what each method hides, and it also means we have no
equal-method measurement of anyone else's gateway — so there is nothing here about who
is fastest. What there is: five axes, a worked reading of one public load test whose
harness is open enough to check line by line, and a six-question
checklist for the next number somebody puts in front of you.

## Why two gateway benchmarks can both be honest and still disagree

"How much latency does a gateway add" sounds like one question. It is at least five, and
each has a defensible answer that lands in a different order of magnitude.

| Axis | One choice | The other choice | Spread it can produce |
|---|---|---|---|
| Upstream | Mock server with fixed latency | Real provider API | Microseconds vs seconds |
| Quantity | Overhead (a subtraction against a baseline) | End-to-end wall clock | The entire upstream time |
| Statistic | P50 | P99 | Tens of times, on the same run |
| Load shape | Sustained constant rate | Burst, long-context, streaming | Different working point entirely |
| Framing | Ratio or percentage | Absolute duration | Hides the denominator either way |

Every cell is a legitimate thing to measure. The problem is that a headline figure
almost never says which cells it came from, and two figures from different cells get
quoted side by side as if they answered the same question.

For the worked example I am using load-test results generated by maximhq's CI for
Bifrost, its open-source LLM gateway
([results file at commit fd42bf5](https://github.com/maximhq/bifrost/blob/fd42bf503660ae75940d351027dcda81976a246f/.github/workflows/scripts/load-test-results.md),
February 2026 — it is no longer on the default branch, so that link is pinned to the
commit), plus the performance pages in its documentation. To be explicit about whose numbers these
are: **every figure attributed to Bifrost here was produced by maximhq, in maximhq's own
environment, testing maximhq's gateway. We did not run any of it, and it says nothing
about our gateway or anyone else's.** What we did do is read the harness that produced
it, which is public, and that turns out to matter more than the figures.

I picked this sample because the method is written down next to the results — rarer than
it should be, and the precondition for everything below. What I found when I followed the
method down into the script is that the numbers do not survive it. That is not a knock on
the project; it is the most useful thing in this post, because it is the case you cannot
detect by reading a benchmark page carefully. You have to read the code, and then you have
to check whether the numbers are internally consistent. By the end of this post you will
have a test for the second part that takes about ten seconds and needs no source access at
all.

It also demonstrates this post's thesis from a single vendor about a single product. The
documentation headlines "less than 15µs added latency per request on average" at 5,000
RPS, and the same page reports 59µs on a t3.medium against 11µs on a t3.xlarge
([Benchmarking — getting started](https://docs.getbifrost.ai/benchmarking/getting-started)).
Hold those two side by side first, because they are the cleanest case available: same
vendor, same page, same statistic, same test — and a 5.4× spread that is purely the
instance type. Nothing is in dispute there, and it already means "Bifrost's overhead"
is not a number.

Now add the CI results file, which reports a mean overhead of 849.31µs at 1000 RPS.
Against the documentation's headline mean of under 15µs, that is a factor of about 57.

It is worth being slow about why, because this is the variable that does the most damage
and the one readers skip. The headline is a **mean**. The CI file's most quotable figure
is a **P50** of 155.78µs. Compare those two directly and you get a gap of about 10× — and
that comparison is invalid, because a mean and a median are answering different
questions about the same requests. The median is the middle request: half were faster,
half slower, and it is completely blind to how bad the slow half got. The mean divides
total time by request count, so every slow request pulls it upward in proportion to how
slow it was. In a distribution with a long tail — which is every latency distribution
worth discussing — the mean sits well above the median, and the distance between them
tracks the tail. Here that distance is the difference between 155.78µs and 849.31µs, a
5.5× disagreement between two summaries of one run, purely from which you pick. (In this
particular row the gap turns out to have a second cause as well, which the next section
gets to. The point about mean versus median holds regardless.)

So the honest comparison is mean against mean, 849.31µs against under 15µs, and the gap
is 57× rather than 10×. Notice which direction the invalid comparison errs: reaching for
the median makes the vendor's two published figures look five times more consistent than
they are. That is the mechanism this whole post is about, caught in one substitution.
The other three variables — instance type, request rate, payload size — move the number
too, and the t3.medium/t3.xlarge pair above shows instance type alone is worth 5.4×. But
they at least announce themselves in the test description. The choice of statistic
usually does not.

## Axis one: a stub measures the gateway, a real upstream measures the provider

Point your load generator at a gateway that talks to a mock server and you are measuring
the gateway's own processing cost. Point it at a real provider and you are mostly
measuring the provider, with the gateway's contribution buried in the noise.

The Bifrost CI run takes the first road, and documents it well enough to follow. It uses
a *mocker* — a stand-in server that returns a canned response after a fixed delay, so the
upstream contributes a known constant instead of a real provider's variable behaviour —
pinned at 1000ms, driven at 1000 requests per second for 30 seconds at roughly 1000
concurrent requests, with a calibration run of Vegeta→Mocker whose time is subtracted
from the Vegeta→Bifrost→Mocker run. What is left after the subtraction is attributed to the
gateway. Under that method the reported overhead is a P50 of 155.78µs and a P99 of
4526.28µs — 0.16ms against 4.5ms — at a 100% success rate. Hold those two figures loosely — the next section
takes the subtraction apart and they do not survive it — but the shape of the method is
the point here. The
documentation says the same thing about its own figures in one line — "all benchmarks
are on mocked OpenAI calls" — which is the disclosure I wish every vendor made in that
position.

Now the other road. OpenRouter publishes a per-provider table for each model: the same
weights, hosted by different companies, with measured P50 latency. For
`openai/gpt-6-astra` on 5 September 2026, the five listed endpoints ranged from 2.04s
(OpenAI Fast) to 6.88s (OpenAI Flex), with Azure (US) at 3.38s and OpenAI's standard
endpoint at 3.67s ([GPT-6 Astra — providers](https://openrouter.ai/openai/gpt-6-astra);
these are rolling three-day P50s and will read differently by the time you check).

Read that table carefully before drawing the obvious conclusion, because most of the
spread is not what it first looks like. Four of those five endpoints are OpenAI's own —
Fast, standard and Flex are service tiers of the same provider, priced from $20/M down
to $5/M output. So the 4.84s range is mostly the tier you bought, not the company you
routed to; the actual cross-host comparison here is Azure (US) at 3.38s against OpenAI
at 3.67s, a difference of under 300ms.

The proportion is still the whole point. Whichever of those two readings you take —
tier selection or host selection — the effect on the median is measured in seconds or
hundreds of milliseconds, and the gateway overhead being argued about is measured in
microseconds. Not the same kind of quantity, and a routing decision that lands on a
better-behaved endpoint has done more for your P50 than any amount of serialisation
tuning. That is not an argument for sloppy gateways. It is an argument for knowing which
number you are optimising.

## Axis two: overhead is a subtraction, and the baseline is where the argument hides

"Overhead" is never measured directly. Something is measured, something else is
measured, and the difference is called overhead. The interesting question about any
overhead figure is therefore not the figure — it is what got subtracted.

The Bifrost run states its subtraction explicitly, which is what let me check it. The
script runs Vegeta against the mocker directly to establish a baseline, runs it again
through the gateway, and subtracts. The detail that matters is on
[lines 563-567 of `load-test.sh`](https://github.com/maximhq/bifrost/blob/fd42bf503660ae75940d351027dcda81976a246f/.github/workflows/scripts/load-test.sh#L563-L567):

```bash
# Subtract calibration per bucket: overhead = through_bifrost - direct_to_mocker
local us_50=$(printf "%.2f" $(echo "scale=4; ($EXTRACTED_50_NS - $CAL_50_NS) / 1000" | bc))
```

The subtraction is done **per percentile bucket**. The calibration run's P50 is
subtracted from the test run's P50, its P99 from the test's P99, and so on. That is not
the same as measuring the overhead of each request and then taking percentiles of those
overheads. The P50 request in the calibration run and the P50 request in the test run
are different requests, from different runs; their difference is not the median overhead
of anything. It is the difference of two order statistics, which is a number, but not a
latency.

You do not need the source to catch this, because the published table contradicts
itself in two places. The minimum overhead is reported as 999926.96µs — 999.93ms, or
essentially the mocker's entire 1000ms artificial delay. No gateway's floor cost is one
second; that is the stub's latency surviving a subtraction that was supposed to remove
it. The mean, 849.31µs, is also larger than both the P90 (408.13µs) and the P95
(636.92µs) — which can happen legitimately under heavy skew, so on its own it is a
warning rather than a verdict. The minimum is the one that settles it: no arrangement of
requests produces a floor of one second and a median of 155.78µs. These seven numbers
are not all describing the same set of requests.

That gives a check anyone can run on a percentile table, with no access to the harness:

> **Verify the numbers still form a coherent distribution.** The minimum must be below
> the P50 and the maximum above the P99 — these are definitional, and a violation is
> proof that the row is not a single population. Then check the mean: it should normally
> sit between the P50 and the P90, and when it does not, look at the maximum before
> concluding anything.

That last caveat is not hedging, and the same report shows why. In the stress table the
mean of 1002.58ms also sits above its P90 of 1001.17ms — but there the maximum is
1286.46ms, far out beyond a very tight body, and a handful of slow requests like that
will drag a mean past the P90 in a perfectly honest measurement. Skew explains it. In
the overhead table there is no such story available: the minimum alone, at 999.93ms, is
already impossible for a figure describing gateway overhead, so the row is disqualified
before the mean is even considered. **A mean above the P90 is a prompt to look harder;
a minimum above the P50, or a floor equal to the stub's own latency, is a verdict.**

The published notes make the same point from the other direction, though not
deliberately. They say the overhead figures ignore mocker jitter and local network
request queuing, and estimate the real-world P99 overhead at approximately 100
microseconds. But the harness itself says the opposite three separate times: its header
comment describes the measured overhead as including "local network hop, JSON
parsing/unparsing, plugins, and mocker jitter", and it logs the same thing at runtime.
Only the line that writes the markdown says "ignores". One of these is wrong, and a
reader with just the results file has no way to know which.

None of this makes the project's engineering suspect — the harness is more transparent
than most, which is the only reason any of it is checkable. What it does is make the
*numbers* unusable as a headline, and put that 100µs estimate on very soft ground: it is
an estimate resting on a subtraction whose output is already incoherent, published in a
file whose own description of what it measured contradicts the code that measured it.

So the minimum bar for an overhead figure is three things, not two: how the baseline was
measured, at which layer the subtraction happened, and whether the result still looks
like a distribution. The first two require disclosure from the publisher. The third you
can check yourself, and it is the one that catches this class of error.

## Axis three: P50 flatters a gateway, P99 is what an agent loop feels

Set the overhead table aside now — we established it cannot be read as a distribution —
and take the gap between its P50 and P99 purely as an illustration of scale: 155.78µs
against 4526.28µs (again, 0.16ms against 4.5ms) is a factor of roughly 29 between two
statistics of the same run — our arithmetic on their published figures. Whatever those particular numbers are worth, a
gap of that order between median and tail is entirely ordinary in real systems, and it
is why the choice of statistic decides the story.

Which one matters depends on what sits on top. For a single interactive request, the
median is a reasonable proxy for what a user feels. For an agent loop it is not: an
agent making twenty sequential tool calls does not experience the median of those
twenty, it experiences their sum, and the slow ones dominate the sum. Chain enough steps
and the P99 stops being a tail event you can wave off and becomes something most tasks
hit at least once.

The same report contains a second lesson that is easy to miss, and this one rests on
firmer ground: the stress-scenario figures are reported end-to-end, with no calibration
subtracted, so they are not affected by anything in the previous section. Same 1000 RPS,
same 30 seconds, but measuring wall-clock time through the 1000ms mocker — and the
spread collapses: P50 1000.64ms, P99 1047.60ms, about 4.7% apart. The gateway's tail did
not disappear; it got diluted by a
fixed one-second upstream that dwarfs it. Same process, same test suite, same day: a 29×
tail spread under one framing and a 1.05× spread under the other. Neither is
manipulated. They answer different questions.

Real production tails come from things a synthetic load test is not simulating: cache
behaviour, queueing at the provider, retries. The longitudinal study in arXiv 2607.13080
— a single-developer case study of coding agents over two 28-day periods, aimed at cost
and defect rates rather than gateway latency — reports a 99.3% prompt-cache hit rate on
its API-based configuration
([Inference Economics of Enterprise Coding Agents](https://arxiv.org/abs/2607.13080)).
Take nothing about gateway overhead from that paper; it does not measure it. Take only
this: at that hit rate, the distribution a real agent workload sees is shaped by which
requests hit cache, and no fixed-latency mock reproduces that shape.

## Axis four: 1000 RPS sustained is not your traffic

This axis gets skipped, because sustained-rate tests are the easiest to run. A constant
1000 RPS for 30 seconds is a specific and fairly gentle load shape:
the process reaches a steady state and stays there. Real traffic arrives in bursts, with
long-context requests that hold buffers open, and increasingly as streaming responses
where the connection stays alive for the length of the generation.

The process statistics in that report make the point without interpretation from me.
During the overhead phase, maximhq's instance averaged 20.7% CPU and 332.8MB RSS; during
the stress phase, 29.9% CPU and 639.5MB RSS, peaking at 789.2MB. Same binary, same rate,
and the memory footprint nearly doubles because the concurrency profile changed. A
gateway characterised at one working point tells you little about its behaviour at
another.

Streaming deserves its own warning. Time-to-first-token and total wall-clock time move
independently, so a benchmark reporting only total duration says nothing about perceived
responsiveness. If you are sizing for a chat UI or an agent that acts on partial output,
a non-streaming benchmark measures the wrong end of the request.

## Axis five: a percentage and a duration hide different things

This one has been running underneath the whole post, so it deserves a name. Every figure
here can be stated two ways, and each framing conceals what the other reveals.

Take the stress numbers from earlier: P50 1000.64ms, P99 1047.60ms. As a ratio that is
4.7% — sounds negligible. As a duration it is 47ms of tail, which on a twenty-step agent
loop is most of a second. Same two numbers. Now run it the other way: the overhead P99 of
4526.28µs sounds alarming as a duration until you notice it sits on top of a 1000ms
upstream, where it is 0.45%. A ratio needs its denominator stated or it means nothing; an
absolute duration needs the total it is a fraction of, or you cannot tell whether it
matters.

The practical form of this: whenever someone gives you a percentage, ask what it is a
percentage of, and whenever someone gives you a duration, ask what it is riding on top
of. Vendors reach for whichever framing flatters — that is not dishonesty so much as
gravity — and the number of gateway pages quoting "less than 1% overhead" without ever
saying one percent of which upstream is high enough to be worth a reflex.

## The six questions

Ask these of any gateway latency figure, including ours whenever we publish one.

1. **Stub or real upstream?** A stubbed run tells you about the gateway's own cost. A
   real-upstream run tells you mostly about the provider. If the answer is missing, you
   cannot tell which of the two you are holding — and on the figures in this post they
   are separated by several orders of magnitude, microseconds against seconds.
2. **Overhead or end-to-end — and if overhead, what was the baseline?** No baseline
   disclosure, no usable number. Not a small number, not a big one: an unreadable one.
3. **Which percentile, over how many requests, for how long?** A 30-second run at 1000
   RPS puts about 30,000 requests behind the number and roughly 300 in the top
   percentile — enough to estimate a P99, thin for a P99.9, and nothing at all for the
   rare stall that will define your worst week. Rate and duration are what vendor pages
   most often omit, so when they are missing: read any figure quoted without a percentile
   as a mean, and look for a published max. In the run above the max is 176968.29µs
   against a P50 of 155.78µs — three orders of magnitude apart, which says more about the
   tail than the P99 does.
4. **What load shape, and does it resemble yours?** Sustained constant rate, burst,
   long-context, streaming — these are different tests. If the published shape is not
   yours, the number is a data point about someone else's system.
5. **Ratio or duration — and what is the denominator?** A percentage without the base it
   is taken from, or a duration without the total it sits inside, is half a number.
6. **Do the numbers still form a distribution?** Min below the median and max above the
   P99, always; mean between the P50 and P90 unless a distant maximum explains the skew.
   This is the only question on the list you can answer without asking anybody anything,
   and as we saw, it is the one that catches errors the other five let through.

A figure answering all six is comparable with another that answers all six the same way.
A figure that answers none is marketing, whoever published it.

## What to do with this

Run the questions against whatever number sent you here. If it fails on question one or
two, you do not have a latency measurement, you have a decoration — go and get the
method, or generate your own.

Generating your own is more tractable than it sounds, and it is why Bifrost has been the
worked example throughout: maximhq ships the benchmark tool it used and documents how to
point it at your own instance, with flags for rate, duration and payload size
([Run Your Own Benchmarks](https://github.com/maximhq/bifrost-benchmarking)).

Which is the note I want to end the analysis on, because it would be easy to read this
post as a hit piece and that would be the wrong lesson entirely. Every criticism above
exists *because* the harness is public. I could read the subtraction, quote the line,
and check the columns against each other. The overwhelming majority of gateway latency
figures — including some quoted far more confidently than these — publish a number, a
logo, and nothing else, and against those there is no analysis to write, only a decision
about whether to believe them. A benchmark you can prove wrong is worth more than one
you cannot examine. That is not a consolation prize; it is the entire standard.

For our part: our gateway is designed for a routing path where provider selection happens
per request, and given axis one, that is the part of the latency budget we think is worth
arguing about — an architectural statement about a system still being built, not a
performance claim, which is why there is no number attached to it. We are preparing a
public benchmark run under an equal method, with the harness and environment published
alongside it, so the figures can be checked against the six questions rather than taken
on our word. No preview of the results here: a number without its method is the thing
this post is about.

---

## Sources

### github.com

- [results file at commit fd42bf5](https://github.com/maximhq/bifrost/blob/fd42bf503660ae75940d351027dcda81976a246f/.github/workflows/scripts/load-test-results.md)
- [lines 563-567 of `load-test.sh`](https://github.com/maximhq/bifrost/blob/fd42bf503660ae75940d351027dcda81976a246f/.github/workflows/scripts/load-test.sh#L563-L567)
- [Run Your Own Benchmarks](https://github.com/maximhq/bifrost-benchmarking)

### docs.getbifrost.ai

- [Benchmarking — getting started](https://docs.getbifrost.ai/benchmarking/getting-started)

### openrouter.ai

- [GPT-6 Astra — providers](https://openrouter.ai/openai/gpt-6-astra)

### arxiv.org

- [Inference Economics of Enterprise Coding Agents](https://arxiv.org/abs/2607.13080)