# FP8 flips 20% of top-1 tokens, and tool calls break first

> A week of teacher-forced logit capture on one set of weights: at 88k context, official FP8 flips about 20% of top-1 tokens against BF16, NVFP4 about 50%. The interesting part is not the ladder. It is that the damage lands on tool calls — wrong interface, corrupted port number, a parameter name quietly truncated — and a routing table keyed on the model name cannot see any of the variables that caused it.

- Published: Aug 22, 2026
- Updated: Aug 24, 2026
- Author: Linden Kern, Chief Scientist
- Tags: quantization, models, routing
- Canonical: https://pirouter.ai/blog/quantization-is-a-product-spec

---
In [an earlier post](/blog/one-weight-many-prices/) we argued that one set of weights
reaches the market as several products with different prices. This week someone measured
the other half of that claim: the same weights, served four ways, also produce different
*behaviour*. At 88k tokens of context, the official FP8 build of a 27B model disagrees
with its BF16 reference on roughly 20% of top-1 token choices. NVFP4 disagrees on about
50%. **The failures are not distributed evenly across a response — they concentrate in
tool calls, which is where an agent workload does its actual work.**

## The measurement, and what it is not

The [Level1Techs study](https://forum.level1techs.com/t/why-your-local-llm-feels-dumber-than-it-is/253917)
uses teacher-forced decoding with logit capture, on an RTX PRO 6000 Blackwell. The first
pass sampled every 32nd token across a 100k-token real workstream — about 3% coverage. The
second pass captured 100% of logits during tool-calling sequences specifically, on the
reasoning that a 3% sample tells you a distribution shifted but not where it hurt.

Before any number: the author's own limits, which are the reason the numbers are worth
reading at all.

> BF16 is a numerical-fidelity reference, not an oracle or correctness label. A quantized
> model can absolutely diverge from BF16 and produce a semantically better answer.

That sentence should survive every summary of this work, including ours. A flip rate is
a measure of *divergence from a reference*, not a defect count. The author is equally
blunt about the 3% pass being too thin to conclude from, and notes that flip rates are
strongly prompt-dependent rather than a smooth function of context length. Anyone
reporting "quantization costs you 50% accuracy" has read a different post than the one
that was written.

There is a related warning aimed at the model cards themselves: a KLD figure means nothing
without the reference checkpoint, runtime environment, evaluation text, calibration data,
context lengths, sampled positions, KL direction and aggregation method. Most published
quantization claims disclose one or two of those.

## The ladder

Top-1 disagreement against the BF16 reference, at 88k context:

| Build | Type | Flips at 88k |
|---|---|---|
| TheHouseOfTheDude INT8 | W8A16 | ~15% |
| Official FP8 | W8A8, E4M3 blocks | ~20% |
| AWQ W4A16 | group size 32 | ~30% |
| NVIDIA NVFP4 | mixed FP8/FP4 | ~50% |

The 50% row is the one that travels, and it is the least interesting. NVFP4 is a
first-generation 4-bit format and the study is one data point against it.

The row that should change a decision is the second one. **Official FP8 — the precision a
great many production deployments run, published by the model's own authors — sits at
about 20% at long context.** Not a community requantization, not an aggressive 4-bit
experiment. The default choice, diverging from the reference on a fifth of its top-1
picks by the time a conversation gets long.

Divergence is not damage. That is what the next section is for.

## Where it actually breaks

Two distinct mechanisms produce tool-call failures in this study, and conflating them
would be a mistake, because they have different fixes.

**Attention kernel.** Holding weights, prompt and seed constant and changing only the
vLLM attention backend produces reproducible, bit-identical differences between runs. In
one case FlashAttention 2 targeted the wrong Cisco interface — selecting
`GigabitEthernet0/1/4` where the correct target was `GigabitEthernet0/0/1.201` — and then
issued remediation commands against it. No quantization involved. Same weights, different
kernel, wrong device.

**KV cache precision.** Quantizing the KV cache while leaving weights and activations
alone: int8 eventually recovered from tool-call divergence, int4 never did. "Never
recovered" is the phrase worth sitting with — not a degraded response, a conversation that
stays broken.

That result needs a boundary drawn around it, added here on 2026-08-24. A llama.cpp
[issue](https://github.com/ggml-org/llama.cpp/issues/21385) measured q4_0 KV cache as
**completely lossless on Qwen3.5 — BLEU 1.000 across ten configurations, token-identical
to f16 at 4× compression**, while GPT-2 (full attention on every layer) scored 0.531 under
the same compression. The proposed explanation is architectural: hybrid models run full
attention on only a fraction of their layers, and the linear-attention layers absorb the
error.

The tempting conclusion — hybrid architectures are safe — does not survive contact with
the model under test here. Qwen3.6-27B **is** a hybrid: the author says so explicitly
("it is still a hybrid model"), with the same three-DeltaNet-to-one-full-attention pattern
across 64 layers. Both studies looked at hybrid models and reached opposite conclusions.

What separates them is what they measured. The llama.cpp figures are token-level agreement
on plain text completion. The Level1Techs failure is a tool call inside a 100k-token
working session, scored on whether the run recovers. **A cache precision can be lossless
for prose and still break structured output**, and neither measurement licenses a claim
about the other's workload. If your traffic is generation, the saved memory is real; if it
carries tool calls, the int4 downside has a documented shape and that shape is
unrecoverable.

![Two-column comparison of tool-call failure mechanisms: attention kernel producing a wrong Cisco interface target, and KV cache precision corrupting a PostgreSQL port, an API parameter and a hostname](/blog/images/quantization-product-spec-failure-modes.png "Fig. 1 — Two mechanisms, one symptom. Neither the attention backend nor the KV cache precision appears in a model name, and both produce a broken tool call. Both cases are tool calls; see the text for why that scope matters. Cases from the [Level1Techs study](https://forum.level1techs.com/t/why-your-local-llm-feels-dumber-than-it-is/253917).")

The third pass, on abliterated fine-tunes of the same base, produced the most legible
failures of the set:

| Failure | What happened |
|---|---|
| PostgreSQL port | `5432` emitted as `543ql` — the final digit replaced by letters |
| API parameter | `page_size=100` collapsed to `size=100` |
| Hostname | `.tenant` written as `.-tenant`, at 99.99% reported confidence |

That last one deserves its own note. The model was not uncertain. It was confident and
wrong, which is precisely the failure mode no downstream validator catches, because
nothing in the output looks hesitant.

Flip rates across those derivatives: Heretic-ARA 0.717% and 1.337% on two prompts,
Huihui Abliterated 0.912% and 1.406%, AEON Ultimate 2.997% and 5.831% with 36 structurally
invalid outputs. The author is careful to note that AEON bundles several techniques, so
the damage cannot be attributed to abliteration alone.

## The counterarguments

The [HN thread](https://news.ycombinator.com/item?id=49402232) produced three objections
worth taking seriously. Two of them narrow the result; one of them widens the problem.

| Objection | Our read |
|---|---|
| `llama.cpp` and `ik_llama.cpp` enforce grammar-constrained generation once a tool call is detected, so this failure mode "is just not possible" there | ✓ Correct, and it matters. Constrained decoding is a real mitigation. It bounds *syntax*, not *choice* — a grammatically perfect call to the wrong interface still passes |
| NVFP4 and AWQ W4A16 are "generally regarded as low quality quants"; IQK/Trellis and EXL3 do better | ⚠ Fair on sample selection, but it does not touch the FP8 row, which is the one we think should change decisions |
| The usual culprit is not quantization at all — it is a missing chat template or wrong sampling parameters | ✓ Agreed, and a useful boundary on this whole post |

That third one, from commenter `anotherCodder`, is the most valuable comment in the
thread, and it argues against over-reading our own headline. Quantization is one variable
among several that a model name does not carry. The same thread has a report that
`llama.cpp`'s Q8_0 outperforms vLLM's FP8 — which is a statement about two runtimes, not
two precisions.

## The same shape, in latency

One [community benchmark](https://www.reddit.com/r/LocalLLaMA/comments/1vvncyh/i_benchmark_dflash_2_pr_build_in_llamacpp_on_qwen/)
of speculative decoding on the same base reports 2.26x on 100 real coding prompts, rising
to 4.68x with an n-gram drafter stacked on top. Self-reported, not independently
reproduced. Take the ratio rather than the absolute: the drafter configuration moves
throughput by roughly 2x on identical weights. Another invisible field.

## A second, independent sample

The clearest corroboration comes from an unrelated investigation. Sergey Kovalov, while
[debugging local model benchmarks](https://sergeykovalov.com/debugging-local-llm-benchmarks/),
traced runaway output to Aider passing `temperature=0` into llama.cpp, where numeric zero
is treated as a present value rather than an omitted default — producing greedy decoding.
One response repeated a 92-line block 32 times, consuming roughly 30,000 output tokens.

He is scrupulous that this is correlation, not a controlled result. What makes the
write-up useful is the record he now keeps per result:

```text
model repo + exact GGUF filename + quantization
llama.cpp revision + build options
endpoint model id, chat template, template arguments
reasoning mode and final-answer extraction
input/context and output limits
seed and complete sampling parameters
benchmark version, tasks, attempts, edit format
raw response, validator checks, latency, token usage
```

That list is the answer to this entire post, arrived at independently. Every line is a
field that changes behaviour and that a model name does not carry.

## What a routing table cannot see

The practical problem is not that quantization degrades quality. It is that the variables
which determine behaviour are invisible at the point where routing decisions get made.

A name-keyed routing table knows one field. The evidence above says at least four matter:
the weights, the quantization, the KV cache precision, and the runtime — and for agent
traffic the failure they produce is not a worse paragraph, it is a call to the wrong
interface with high confidence.

Most of that is not observable from outside a host today, and pretending otherwise would
be worse than admitting it. Precision is sometimes declared. KV cache precision and
attention backend almost never are. Our own catalog treats declared precision as
first-class, filterable metadata and surfaces undeclared values as unknown rather than
guessing — a description of how it is designed, not a claim that hosts publish what we
would like them to.

Which leaves the self-service check, the same one this study performs: run your own
prompts against a reference you trust, and measure the divergence on the calls that
matter. Not a benchmark score — your tool calls, at the context length you actually use.
The study's most reproducible finding is that this is worth doing, because two deployments
of one model can disagree with each other half the time, and the model card will not
mention it.

---

## Sources

### forum.level1techs.com

- [Level1Techs study](https://forum.level1techs.com/t/why-your-local-llm-feels-dumber-than-it-is/253917)

### github.com

- [issue](https://github.com/ggml-org/llama.cpp/issues/21385)

### news.ycombinator.com

- [HN thread](https://news.ycombinator.com/item?id=49402232)

### reddit.com

- [community benchmark](https://www.reddit.com/r/LocalLLaMA/comments/1vvncyh/i_benchmark_dflash_2_pr_build_in_llamacpp_on_qwen/)

### sergeykovalov.com

- [debugging local model benchmarks](https://sergeykovalov.com/debugging-local-llm-benchmarks/)