How to measure the latency an LLM gateway adds: six questions to ask before you believe a number

You cannot compare two LLM gateway latency numbers unless you know five things about each: whether the upstream was stubbed or real, whether the number is an overhead subtraction or an end-to-end measurement, which percentile it is, what load shape produced it, and whether it is framed as a ratio or a duration. Change any one of those and the same gateway on the same hardware will report a figure that differs by orders of magnitude — honestly, with nobody lying. That is why the benchmarks you find disagree. And there is a sixth thing, which no publisher has to tell you because you can check it yourself: whether the numbers in front of you are even mutually consistent. Below, one published table fails that check.

This post is not a ranking. We build a gateway, which means we know exactly how these numbers are produced and what each method hides, and it also means we have no equal-method measurement of anyone else’s gateway — so there is nothing here about who is fastest. What there is: five axes, a worked reading of one public load test whose harness is open enough to check line by line, and a six-question checklist for the next number somebody puts in front of you.

Why two gateway benchmarks can both be honest and still disagree

“How much latency does a gateway add” sounds like one question. It is at least five, and each has a defensible answer that lands in a different order of magnitude.

Every cell is a legitimate thing to measure. The problem is that a headline figure almost never says which cells it came from, and two figures from different cells get quoted side by side as if they answered the same question.

For the worked example I am using load-test results generated by maximhq’s CI for Bifrost, its open-source LLM gateway (results file at commit fd42bf5, February 2026 — it is no longer on the default branch, so that link is pinned to the commit), plus the performance pages in its documentation. To be explicit about whose numbers these are: every figure attributed to Bifrost here was produced by maximhq, in maximhq’s own environment, testing maximhq’s gateway. We did not run any of it, and it says nothing about our gateway or anyone else’s. What we did do is read the harness that produced it, which is public, and that turns out to matter more than the figures.

I picked this sample because the method is written down next to the results — rarer than it should be, and the precondition for everything below. What I found when I followed the method down into the script is that the numbers do not survive it. That is not a knock on the project; it is the most useful thing in this post, because it is the case you cannot detect by reading a benchmark page carefully. You have to read the code, and then you have to check whether the numbers are internally consistent. By the end of this post you will have a test for the second part that takes about ten seconds and needs no source access at all.

It also demonstrates this post’s thesis from a single vendor about a single product. The documentation headlines “less than 15µs added latency per request on average” at 5,000 RPS, and the same page reports 59µs on a t3.medium against 11µs on a t3.xlarge (Benchmarking — getting started). Hold those two side by side first, because they are the cleanest case available: same vendor, same page, same statistic, same test — and a 5.4× spread that is purely the instance type. Nothing is in dispute there, and it already means “Bifrost’s overhead” is not a number.

Now add the CI results file, which reports a mean overhead of 849.31µs at 1000 RPS. Against the documentation’s headline mean of under 15µs, that is a factor of about 57.

It is worth being slow about why, because this is the variable that does the most damage and the one readers skip. The headline is a mean. The CI file’s most quotable figure is a P50 of 155.78µs. Compare those two directly and you get a gap of about 10× — and that comparison is invalid, because a mean and a median are answering different questions about the same requests. The median is the middle request: half were faster, half slower, and it is completely blind to how bad the slow half got. The mean divides total time by request count, so every slow request pulls it upward in proportion to how slow it was. In a distribution with a long tail — which is every latency distribution worth discussing — the mean sits well above the median, and the distance between them tracks the tail. Here that distance is the difference between 155.78µs and 849.31µs, a 5.5× disagreement between two summaries of one run, purely from which you pick. (In this particular row the gap turns out to have a second cause as well, which the next section gets to. The point about mean versus median holds regardless.)

So the honest comparison is mean against mean, 849.31µs against under 15µs, and the gap is 57× rather than 10×. Notice which direction the invalid comparison errs: reaching for the median makes the vendor’s two published figures look five times more consistent than they are. That is the mechanism this whole post is about, caught in one substitution. The other three variables — instance type, request rate, payload size — move the number too, and the t3.medium/t3.xlarge pair above shows instance type alone is worth 5.4×. But they at least announce themselves in the test description. The choice of statistic usually does not.

Axis one: a stub measures the gateway, a real upstream measures the provider

Point your load generator at a gateway that talks to a mock server and you are measuring the gateway’s own processing cost. Point it at a real provider and you are mostly measuring the provider, with the gateway’s contribution buried in the noise.

The Bifrost CI run takes the first road, and documents it well enough to follow. It uses a mocker — a stand-in server that returns a canned response after a fixed delay, so the upstream contributes a known constant instead of a real provider’s variable behaviour — pinned at 1000ms, driven at 1000 requests per second for 30 seconds at roughly 1000 concurrent requests, with a calibration run of Vegeta→Mocker whose time is subtracted from the Vegeta→Bifrost→Mocker run. What is left after the subtraction is attributed to the gateway. Under that method the reported overhead is a P50 of 155.78µs and a P99 of 4526.28µs — 0.16ms against 4.5ms — at a 100% success rate. Hold those two figures loosely — the next section takes the subtraction apart and they do not survive it — but the shape of the method is the point here. The documentation says the same thing about its own figures in one line — “all benchmarks are on mocked OpenAI calls” — which is the disclosure I wish every vendor made in that position.

Now the other road. OpenRouter publishes a per-provider table for each model: the same weights, hosted by different companies, with measured P50 latency. For openai/gpt-6-astra on 5 September 2026, the five listed endpoints ranged from 2.04s (OpenAI Fast) to 6.88s (OpenAI Flex), with Azure (US) at 3.38s and OpenAI’s standard endpoint at 3.67s (GPT-6 Astra — providers; these are rolling three-day P50s and will read differently by the time you check).

Read that table carefully before drawing the obvious conclusion, because most of the spread is not what it first looks like. Four of those five endpoints are OpenAI’s own — Fast, standard and Flex are service tiers of the same provider, priced from $20/M down to $5/M output. So the 4.84s range is mostly the tier you bought, not the company you routed to; the actual cross-host comparison here is Azure (US) at 3.38s against OpenAI at 3.67s, a difference of under 300ms.

The proportion is still the whole point. Whichever of those two readings you take — tier selection or host selection — the effect on the median is measured in seconds or hundreds of milliseconds, and the gateway overhead being argued about is measured in microseconds. Not the same kind of quantity, and a routing decision that lands on a better-behaved endpoint has done more for your P50 than any amount of serialisation tuning. That is not an argument for sloppy gateways. It is an argument for knowing which number you are optimising.

Axis two: overhead is a subtraction, and the baseline is where the argument hides

“Overhead” is never measured directly. Something is measured, something else is measured, and the difference is called overhead. The interesting question about any overhead figure is therefore not the figure — it is what got subtracted.

The Bifrost run states its subtraction explicitly, which is what let me check it. The script runs Vegeta against the mocker directly to establish a baseline, runs it again through the gateway, and subtracts. The detail that matters is on lines 563-567 of load-test.sh:

# Subtract calibration per bucket: overhead = through_bifrost - direct_to_mocker
local us_50=$(printf "%.2f" $(echo "scale=4; ($EXTRACTED_50_NS - $CAL_50_NS) / 1000" | bc))

The subtraction is done per percentile bucket. The calibration run’s P50 is subtracted from the test run’s P50, its P99 from the test’s P99, and so on. That is not the same as measuring the overhead of each request and then taking percentiles of those overheads. The P50 request in the calibration run and the P50 request in the test run are different requests, from different runs; their difference is not the median overhead of anything. It is the difference of two order statistics, which is a number, but not a latency.

You do not need the source to catch this, because the published table contradicts itself in two places. The minimum overhead is reported as 999926.96µs — 999.93ms, or essentially the mocker’s entire 1000ms artificial delay. No gateway’s floor cost is one second; that is the stub’s latency surviving a subtraction that was supposed to remove it. The mean, 849.31µs, is also larger than both the P90 (408.13µs) and the P95 (636.92µs) — which can happen legitimately under heavy skew, so on its own it is a warning rather than a verdict. The minimum is the one that settles it: no arrangement of requests produces a floor of one second and a median of 155.78µs. These seven numbers are not all describing the same set of requests.

That gives a check anyone can run on a percentile table, with no access to the harness:

Verify the numbers still form a coherent distribution. The minimum must be below the P50 and the maximum above the P99 — these are definitional, and a violation is proof that the row is not a single population. Then check the mean: it should normally sit between the P50 and the P90, and when it does not, look at the maximum before concluding anything.

That last caveat is not hedging, and the same report shows why. In the stress table the mean of 1002.58ms also sits above its P90 of 1001.17ms — but there the maximum is 1286.46ms, far out beyond a very tight body, and a handful of slow requests like that will drag a mean past the P90 in a perfectly honest measurement. Skew explains it. In the overhead table there is no such story available: the minimum alone, at 999.93ms, is already impossible for a figure describing gateway overhead, so the row is disqualified before the mean is even considered. A mean above the P90 is a prompt to look harder; a minimum above the P50, or a floor equal to the stub’s own latency, is a verdict.

The published notes make the same point from the other direction, though not deliberately. They say the overhead figures ignore mocker jitter and local network request queuing, and estimate the real-world P99 overhead at approximately 100 microseconds. But the harness itself says the opposite three separate times: its header comment describes the measured overhead as including “local network hop, JSON parsing/unparsing, plugins, and mocker jitter”, and it logs the same thing at runtime. Only the line that writes the markdown says “ignores”. One of these is wrong, and a reader with just the results file has no way to know which.

None of this makes the project’s engineering suspect — the harness is more transparent than most, which is the only reason any of it is checkable. What it does is make the numbers unusable as a headline, and put that 100µs estimate on very soft ground: it is an estimate resting on a subtraction whose output is already incoherent, published in a file whose own description of what it measured contradicts the code that measured it.

So the minimum bar for an overhead figure is three things, not two: how the baseline was measured, at which layer the subtraction happened, and whether the result still looks like a distribution. The first two require disclosure from the publisher. The third you can check yourself, and it is the one that catches this class of error.

Axis three: P50 flatters a gateway, P99 is what an agent loop feels

Set the overhead table aside now — we established it cannot be read as a distribution — and take the gap between its P50 and P99 purely as an illustration of scale: 155.78µs against 4526.28µs (again, 0.16ms against 4.5ms) is a factor of roughly 29 between two statistics of the same run — our arithmetic on their published figures. Whatever those particular numbers are worth, a gap of that order between median and tail is entirely ordinary in real systems, and it is why the choice of statistic decides the story.

Which one matters depends on what sits on top. For a single interactive request, the median is a reasonable proxy for what a user feels. For an agent loop it is not: an agent making twenty sequential tool calls does not experience the median of those twenty, it experiences their sum, and the slow ones dominate the sum. Chain enough steps and the P99 stops being a tail event you can wave off and becomes something most tasks hit at least once.

The same report contains a second lesson that is easy to miss, and this one rests on firmer ground: the stress-scenario figures are reported end-to-end, with no calibration subtracted, so they are not affected by anything in the previous section. Same 1000 RPS, same 30 seconds, but measuring wall-clock time through the 1000ms mocker — and the spread collapses: P50 1000.64ms, P99 1047.60ms, about 4.7% apart. The gateway’s tail did not disappear; it got diluted by a fixed one-second upstream that dwarfs it. Same process, same test suite, same day: a 29× tail spread under one framing and a 1.05× spread under the other. Neither is manipulated. They answer different questions.

Real production tails come from things a synthetic load test is not simulating: cache behaviour, queueing at the provider, retries. The longitudinal study in arXiv 2607.13080 — a single-developer case study of coding agents over two 28-day periods, aimed at cost and defect rates rather than gateway latency — reports a 99.3% prompt-cache hit rate on its API-based configuration (Inference Economics of Enterprise Coding Agents). Take nothing about gateway overhead from that paper; it does not measure it. Take only this: at that hit rate, the distribution a real agent workload sees is shaped by which requests hit cache, and no fixed-latency mock reproduces that shape.

Axis four: 1000 RPS sustained is not your traffic

This axis gets skipped, because sustained-rate tests are the easiest to run. A constant 1000 RPS for 30 seconds is a specific and fairly gentle load shape: the process reaches a steady state and stays there. Real traffic arrives in bursts, with long-context requests that hold buffers open, and increasingly as streaming responses where the connection stays alive for the length of the generation.

The process statistics in that report make the point without interpretation from me. During the overhead phase, maximhq’s instance averaged 20.7% CPU and 332.8MB RSS; during the stress phase, 29.9% CPU and 639.5MB RSS, peaking at 789.2MB. Same binary, same rate, and the memory footprint nearly doubles because the concurrency profile changed. A gateway characterised at one working point tells you little about its behaviour at another.

Streaming deserves its own warning. Time-to-first-token and total wall-clock time move independently, so a benchmark reporting only total duration says nothing about perceived responsiveness. If you are sizing for a chat UI or an agent that acts on partial output, a non-streaming benchmark measures the wrong end of the request.

Axis five: a percentage and a duration hide different things

This one has been running underneath the whole post, so it deserves a name. Every figure here can be stated two ways, and each framing conceals what the other reveals.

Take the stress numbers from earlier: P50 1000.64ms, P99 1047.60ms. As a ratio that is 4.7% — sounds negligible. As a duration it is 47ms of tail, which on a twenty-step agent loop is most of a second. Same two numbers. Now run it the other way: the overhead P99 of 4526.28µs sounds alarming as a duration until you notice it sits on top of a 1000ms upstream, where it is 0.45%. A ratio needs its denominator stated or it means nothing; an absolute duration needs the total it is a fraction of, or you cannot tell whether it matters.

The practical form of this: whenever someone gives you a percentage, ask what it is a percentage of, and whenever someone gives you a duration, ask what it is riding on top of. Vendors reach for whichever framing flatters — that is not dishonesty so much as gravity — and the number of gateway pages quoting “less than 1% overhead” without ever saying one percent of which upstream is high enough to be worth a reflex.

The six questions

Ask these of any gateway latency figure, including ours whenever we publish one.

1. Stub or real upstream? A stubbed run tells you about the gateway’s own cost. A real-upstream run tells you mostly about the provider. If the answer is missing, you cannot tell which of the two you are holding — and on the figures in this post they are separated by several orders of magnitude, microseconds against seconds.
2. Overhead or end-to-end — and if overhead, what was the baseline? No baseline disclosure, no usable number. Not a small number, not a big one: an unreadable one.
3. Which percentile, over how many requests, for how long? A 30-second run at 1000 RPS puts about 30,000 requests behind the number and roughly 300 in the top percentile — enough to estimate a P99, thin for a P99.9, and nothing at all for the rare stall that will define your worst week. Rate and duration are what vendor pages most often omit, so when they are missing: read any figure quoted without a percentile as a mean, and look for a published max. In the run above the max is 176968.29µs against a P50 of 155.78µs — three orders of magnitude apart, which says more about the tail than the P99 does.
4. What load shape, and does it resemble yours? Sustained constant rate, burst, long-context, streaming — these are different tests. If the published shape is not yours, the number is a data point about someone else’s system.
5. Ratio or duration — and what is the denominator? A percentage without the base it is taken from, or a duration without the total it sits inside, is half a number.
6. Do the numbers still form a distribution? Min below the median and max above the P99, always; mean between the P50 and P90 unless a distant maximum explains the skew. This is the only question on the list you can answer without asking anybody anything, and as we saw, it is the one that catches errors the other five let through.

A figure answering all six is comparable with another that answers all six the same way. A figure that answers none is marketing, whoever published it.

What to do with this

Run the questions against whatever number sent you here. If it fails on question one or two, you do not have a latency measurement, you have a decoration — go and get the method, or generate your own.

Generating your own is more tractable than it sounds, and it is why Bifrost has been the worked example throughout: maximhq ships the benchmark tool it used and documents how to point it at your own instance, with flags for rate, duration and payload size (Run Your Own Benchmarks).

Which is the note I want to end the analysis on, because it would be easy to read this post as a hit piece and that would be the wrong lesson entirely. Every criticism above exists because the harness is public. I could read the subtraction, quote the line, and check the columns against each other. The overwhelming majority of gateway latency figures — including some quoted far more confidently than these — publish a number, a logo, and nothing else, and against those there is no analysis to write, only a decision about whether to believe them. A benchmark you can prove wrong is worth more than one you cannot examine. That is not a consolation prize; it is the entire standard.

For our part: our gateway is designed for a routing path where provider selection happens per request, and given axis one, that is the part of the latency budget we think is worth arguing about — an architectural statement about a system still being built, not a performance claim, which is why there is no number attached to it. We are preparing a public benchmark run under an equal method, with the harness and environment published alongside it, so the figures can be checked against the six questions rather than taken on our word. No preview of the results here: a number without its method is the thing this post is about.