FP8 flips 20% of top-1 tokens, and tool calls break first

In an earlier post we argued that one set of weights reaches the market as several products with different prices. This week someone measured the other half of that claim: the same weights, served four ways, also produce different behaviour. At 88k tokens of context, the official FP8 build of a 27B model disagrees with its BF16 reference on roughly 20% of top-1 token choices. NVFP4 disagrees on about 50%. The failures are not distributed evenly across a response — they concentrate in tool calls, which is where an agent workload does its actual work.

The measurement, and what it is not

The Level1Techs study uses teacher-forced decoding with logit capture, on an RTX PRO 6000 Blackwell. The first pass sampled every 32nd token across a 100k-token real workstream — about 3% coverage. The second pass captured 100% of logits during tool-calling sequences specifically, on the reasoning that a 3% sample tells you a distribution shifted but not where it hurt.

Before any number: the author’s own limits, which are the reason the numbers are worth reading at all.

BF16 is a numerical-fidelity reference, not an oracle or correctness label. A quantized model can absolutely diverge from BF16 and produce a semantically better answer.

That sentence should survive every summary of this work, including ours. A flip rate is a measure of divergence from a reference, not a defect count. The author is equally blunt about the 3% pass being too thin to conclude from, and notes that flip rates are strongly prompt-dependent rather than a smooth function of context length. Anyone reporting “quantization costs you 50% accuracy” has read a different post than the one that was written.

There is a related warning aimed at the model cards themselves: a KLD figure means nothing without the reference checkpoint, runtime environment, evaluation text, calibration data, context lengths, sampled positions, KL direction and aggregation method. Most published quantization claims disclose one or two of those.

The ladder

Top-1 disagreement against the BF16 reference, at 88k context:

The 50% row is the one that travels, and it is the least interesting. NVFP4 is a first-generation 4-bit format and the study is one data point against it.

The row that should change a decision is the second one. Official FP8 — the precision a great many production deployments run, published by the model’s own authors — sits at about 20% at long context. Not a community requantization, not an aggressive 4-bit experiment. The default choice, diverging from the reference on a fifth of its top-1 picks by the time a conversation gets long.

Divergence is not damage. That is what the next section is for.

Where it actually breaks

Two distinct mechanisms produce tool-call failures in this study, and conflating them would be a mistake, because they have different fixes.

Attention kernel. Holding weights, prompt and seed constant and changing only the vLLM attention backend produces reproducible, bit-identical differences between runs. In one case FlashAttention 2 targeted the wrong Cisco interface — selecting GigabitEthernet0/1/4 where the correct target was GigabitEthernet0/0/1.201 — and then issued remediation commands against it. No quantization involved. Same weights, different kernel, wrong device.

KV cache precision. Quantizing the KV cache while leaving weights and activations alone: int8 eventually recovered from tool-call divergence, int4 never did. “Never recovered” is the phrase worth sitting with — not a degraded response, a conversation that stays broken.

That result needs a boundary drawn around it, added here on 2026-08-24. A llama.cpp issue measured q4_0 KV cache as completely lossless on Qwen3.5 — BLEU 1.000 across ten configurations, token-identical to f16 at 4× compression, while GPT-2 (full attention on every layer) scored 0.531 under the same compression. The proposed explanation is architectural: hybrid models run full attention on only a fraction of their layers, and the linear-attention layers absorb the error.

The tempting conclusion — hybrid architectures are safe — does not survive contact with the model under test here. Qwen3.6-27B is a hybrid: the author says so explicitly (“it is still a hybrid model”), with the same three-DeltaNet-to-one-full-attention pattern across 64 layers. Both studies looked at hybrid models and reached opposite conclusions.

What separates them is what they measured. The llama.cpp figures are token-level agreement on plain text completion. The Level1Techs failure is a tool call inside a 100k-token working session, scored on whether the run recovers. A cache precision can be lossless for prose and still break structured output, and neither measurement licenses a claim about the other’s workload. If your traffic is generation, the saved memory is real; if it carries tool calls, the int4 downside has a documented shape and that shape is unrecoverable.

The third pass, on abliterated fine-tunes of the same base, produced the most legible failures of the set:

That last one deserves its own note. The model was not uncertain. It was confident and wrong, which is precisely the failure mode no downstream validator catches, because nothing in the output looks hesitant.

Flip rates across those derivatives: Heretic-ARA 0.717% and 1.337% on two prompts, Huihui Abliterated 0.912% and 1.406%, AEON Ultimate 2.997% and 5.831% with 36 structurally invalid outputs. The author is careful to note that AEON bundles several techniques, so the damage cannot be attributed to abliteration alone.

The counterarguments

The HN thread produced three objections worth taking seriously. Two of them narrow the result; one of them widens the problem.

That third one, from commenter anotherCodder, is the most valuable comment in the thread, and it argues against over-reading our own headline. Quantization is one variable among several that a model name does not carry. The same thread has a report that llama.cpp’s Q8_0 outperforms vLLM’s FP8 — which is a statement about two runtimes, not two precisions.

The same shape, in latency

One community benchmark of speculative decoding on the same base reports 2.26x on 100 real coding prompts, rising to 4.68x with an n-gram drafter stacked on top. Self-reported, not independently reproduced. Take the ratio rather than the absolute: the drafter configuration moves throughput by roughly 2x on identical weights. Another invisible field.

A second, independent sample

The clearest corroboration comes from an unrelated investigation. Sergey Kovalov, while debugging local model benchmarks, traced runaway output to Aider passing temperature=0 into llama.cpp, where numeric zero is treated as a present value rather than an omitted default — producing greedy decoding. One response repeated a 92-line block 32 times, consuming roughly 30,000 output tokens.

He is scrupulous that this is correlation, not a controlled result. What makes the write-up useful is the record he now keeps per result:

model repo + exact GGUF filename + quantization
llama.cpp revision + build options
endpoint model id, chat template, template arguments
reasoning mode and final-answer extraction
input/context and output limits
seed and complete sampling parameters
benchmark version, tasks, attempts, edit format
raw response, validator checks, latency, token usage

That list is the answer to this entire post, arrived at independently. Every line is a field that changes behaviour and that a model name does not carry.

What a routing table cannot see

The practical problem is not that quantization degrades quality. It is that the variables which determine behaviour are invisible at the point where routing decisions get made.

A name-keyed routing table knows one field. The evidence above says at least four matter: the weights, the quantization, the KV cache precision, and the runtime — and for agent traffic the failure they produce is not a worse paragraph, it is a call to the wrong interface with high confidence.

Most of that is not observable from outside a host today, and pretending otherwise would be worse than admitting it. Precision is sometimes declared. KV cache precision and attention backend almost never are. Our own catalog treats declared precision as first-class, filterable metadata and surfaces undeclared values as unknown rather than guessing — a description of how it is designed, not a claim that hosts publish what we would like them to.

Which leaves the self-service check, the same one this study performs: run your own prompts against a reference you trust, and measure the divergence on the calls that matter. Not a benchmark score — your tool calls, at the context length you actually use. The study’s most reproducible finding is that this is worth doing, because two deployments of one model can disagree with each other half the time, and the model card will not mention it.