Why your IQ2 GGUF is the same size as IQ4: the 256-block fallback in llama-quantize

A GGUF filename records what llama-quantize was asked to make. It does not record what the quantizer was able to make. Every k-quant and i-quant type packs weights in blocks of 256, so a tensor whose first dimension is not a multiple of 256 cannot hold one of those types at all; when that happens, llama.cpp substitutes a 32-block type of roughly 4.5 bits per weight, prints a warning into the quantize log, and writes the file under the name that was requested. On 2026-08-29 we read the tensor tables of three public repositories. Unsloth’s Qwen3.8-Flash-Next-UD-IQ1_S, labelled 1.56 bits per weight, measures 3.28. All four IQ2 files of bartowski’s Nemotron-3.5-Lightning measure 4.58 — one density under four names. A dense Qwen3.8-27B file at the same label measures 2.16, which is what the label says.

This post is the third in a series. One weight, many prices argued that the product you call is a tuple — weights, quantization, context window, version — not a name. FP8 flips 20% of top-1 tokens showed that the quantization field in that tuple changes behaviour, not just size. This one is about a smaller and more uncomfortable fact: the quantization field itself can be wrong, honestly, with nobody lying. The rest of the post goes through the arithmetic, the code, a census someone else ran, the numbers we reproduced, and what any of it should change about how you pick a file or a host.

What “IQ2” promises, and the one arithmetic fact that can break it

A quantization type in GGUF is a block format: Q4_0 stores 32 weights per block with a shared scale, IQ2_XXS stores 256 per block with a codebook. Bits per weight is the block’s byte size times eight, divided by its block size, which gives the ladder every download page implies:

Figures from llama.cpp’s GGML_QUANT_SIZES, as tabulated in the ggufaudit README — Bits-per-weight reference table.

Now the arithmetic fact. A block of 256 has to fit an integer number of times into a tensor’s first dimension, ne[0], which is the row width the quantizer walks along. Every type on the left half of that table requires ne[0] % 256 == 0. Most dense transformers satisfy it everywhere — hidden sizes of 4096, 5120, 8192 and feed-forward widths that are multiples of 1024 are the norm — which is why nobody thought about it for years. The newer hybrid and MoE architectures do not always satisfy it. Nemotron 3.5 Lightning has an embedding width of 2688 and expert widths of 1856 and 3712. Qwen3.8-Flash-Next has expert and dense feed-forward widths of 640, 320 and 160. None of those is a multiple of 256, so no k-quant or i-quant can legally be written to those tensors, whatever the recipe asked for.

Where the fallback lives: tensor_type_fallback() in src/llama-quant.cpp

What llama.cpp does about it is not hidden; it is a named function. In src/llama-quant.cpp on master (HEAD 6c84c7d5d8, 2026-08-27, as read on 2026-08-29), tensor_type_fallback() takes the type the recipe chose, checks ncols % qk_k, and if the check fails, logs a warning and switches on the type:

• IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS become IQ4_NL (4.50 bpw)
• Q2_0, Q2_K, Q3_K, TQ1_0, TQ2_0 become Q4_0 (4.50 bpw)
• Q4_K becomes Q5_0 (5.50), Q5_K becomes Q5_1 (6.00), Q6_K becomes Q8_0 (8.50)
• anything whose width is not even a multiple of 32 becomes F16, with a comment that this “is very rare”

The function is called once per tensor, right after the recipe’s own type selection: new_type = tensor_type_fallback(qs, tensor, new_type);. It increments a counter, and at the end of the run the quantizer prints one summary line. The two warnings look like this — the first is per tensor, the second is the summary, both reproduced from llama.cpp issue #26616, where the reporter ran the quantizer on a tiny model to show the effect:

warning: token_embd.weight - ncols 288 not divisible by 256 (required for type q4_K) -> falling back to q5_0
[... 37 such lines total ...]
llama_model_quantize_impl: WARNING: 37 of 57 tensor(s) required fallback quantization

Then the process exits 0. Nothing about the substitution is written into the file: the general.file_type metadata still names the requested type, the filename is whatever the maker chose, and the per-tensor types in the GGUF header are the only record. The counter itself is declared, incremented, logged and reset. It is not persisted.

Two pieces of history matter here, because they decide who this is a criticism of. The fallback was added deliberately in PR #3747 — Allow quantizing k-quants to fall back when tensor size incompatible, merged in October 2023, so that models with awkward widths could be quantized at all rather than failing; the author wrote that the fallback choices were picked “trying to maintain quality over size.” That is a defensible choice for a tool. And the request to make it opt-out — a --no-fallback flag so the quantizer fails before spending compute — is open as issue #26616, filed 2026-08-05 by someone who asked for a Q4_K_M, expected about 18 GB, and got 24.5 GB: roughly 6.2 bits per weight from a “4-bit” request. Their write-up puts the cost of the difference in the only unit that settles an argument — the two AWS GPU instances they were choosing between had 48 GB and 24 GB of VRAM, at about $1,360 and $588 a month. A file that will not fit the smaller one moves you to the larger one. The warning was in the log the whole time. Almost nobody who downloads a GGUF ever sees one.

The census: 443 files, 64 affected, one mechanism

On 2026-08-28 a post on r/LocalLLaMA by u/Daxfortuna put a number on how often this happens in published files. The author wrote a single-file Python tool that reads only a GGUF’s header and tensor table — locally, or from a Hugging Face repository through HTTP range requests, a few megabytes per file — and compares the per-tensor types against the label. Over 443 quantized files in 25 repositories, 64 had dimension-forced substitutions covering at least 1% of parameters. Those are the author’s figures, from the ggufaudit census dated 2026-08-27; we reproduce three of the repositories below, not all 25.

The census is worth reading for its clean rows as much as its dirty ones. MiniMax-M2.1 in 23 rungs including a real IQ1_S, zero forced tensors; Ornith-1.5 across 27 files, zero; the dense Llama and Qwen controls, zero. And the affected repositories cluster: three different makers — bartowski, lmstudio-community and Unsloth — published Nemotron-3.5-Lightning, and all three came out at 98.9% to 99.0% of parameters forced, because all three ran the same quantizer on the same tensor widths. The author’s own sentence is the right one: “Every maker with an affected repo in my census also has a clean one using the same pipeline. The model’s tensor dimensions decide this, not the maker.”

I want to be plain about that, because “mislabeled” is a loaded word and the tool uses it. Nobody in this story falsified anything. The recipe was valid, the quantizer succeeded, the output was published under the standard name the ecosystem uses, and the only place the substitution was ever announced is a log file that is not part of the upload. The tool’s own FAQ says the fix “belongs in tooling (and labeling), not in blaming individual quantizers,” and the census credits a Hugging Face discussion from 2026-08-12 — Low-bit quants fall back to IQ4_NL on this arch — in which another quantization publisher reported the same mechanism on Unsloth’s repo and noted it “hits everyone publishing for this model, us included until we went looking.” This is a property of the toolchain meeting a new generation of architectures. The question for the rest of us is only how to see it.

Reproducing it: four Nemotron IQ2 files, 4.58 bpw each

A census is a claim until someone else runs it, so we did, on 2026-08-29 between 05:24 and 05:33 UTC. The tool is one Python file with no dependencies beyond the standard library; before running it I read it end to end — its network access is limited to the Hugging Face API and range reads of /resolve/main/<file>, it writes nothing and spawns nothing — and ran its built-in self test, which constructs a synthetic GGUF in memory and checks the parser and the fallback detector against known-forced and known-by-design tensors.

curl -sL -o ggufaudit.py https://raw.githubusercontent.com/JoshBolding/ggufaudit/main/ggufaudit.py
python3 ggufaudit.py --selfcheck          # selfcheck: ALL PASS
python3 ggufaudit.py hf://bartowski/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF --quant IQ2

The third line fetched 12 MiB of headers per file in two range requests each and returned this summary (our run, 2026-08-29):

Four files, a claimed range of 2.06 to 2.56 bits, and 0.02 GiB — about 21 MB — of difference among them. The per-class breakdown says where the density comes from. This is the IQ2_M file, trimmed to the rows that matter:

  label: metadata=IQ2_M (general.file_type=29)  filename=IQ2_M
  tensors: 417   params: 32.91B   bytes: 17.55 GiB
  nominal: IQ2_M -> base type IQ2_S = 2.562 bpw     actual overall: 4.581 bpw   (+79% vs nominal)

  class                  tensors  params%   bytes%     bpw   types
  routed experts              48    93.1%    91.5%    4.50   IQ4_NL(46) Q4_0(2)

  DIMENSION-FORCED SUBSTITUTIONS: 142 tensors (llama-quantize fallback; the label does not describe these)
    routed experts           23 tensors -> IQ4_NL  (4.50 bpw)  ncols=1856 (%256=64)   44.6% of params
    routed experts           23 tensors -> IQ4_NL  (4.50 bpw)  ncols=2688 (%256=128)  44.6% of params

The routed experts are 93% of the model. Their widths are 1856 and 2688, which leave remainders of 64 and 128 when divided by 256, so every one of them went to IQ4_NL. The few tensors that did receive the recipe’s intended 2-bit types are the ones with widths that happen to divide — a handful inside the state-space layers — and they are too small to move the average. The label described a recipe that could be applied to about 1% of the weights.

Qwen3.8-Flash-Next: nine of eleven files, and what 51.9% actually measures

The second repository is the one that matters to more readers this week. unsloth/Qwen3.8-Flash-Next-GGUF is the file set behind most of the “it runs on my 128 GB machine” reports since the model’s release on 2026-08-26, and the dynamic recipe behind it is the same one we read carefully in One weight, many prices. We audited the entire repository — eleven model files plus two projector files, thirteen headers, 05:26 to 05:28 UTC — and the tool flagged nine of eleven:

Read the verdict column as a label check, not a quality score: ✗ means the measured density is materially above what the filename claims, ⚠ means forced tensors are present without moving the file’s average, ✓ means label and contents agree. A ✗ file is not broken and not worse — it holds more bits than its name implies. What it costs you is the disk and memory you budgeted from the label, and the smaller file you thought you were choosing.

Measured bpw is our run (2026-08-29); the tool reports sizes in GiB (67.56 GiB for UD-IQ1_S) and the GB column is the decimal figure shown on Hugging Face and in Unsloth’s Qwen3.8-Flash-Next guide, which agree to the first decimal. The verdict column is ours, not the tool’s: ggufaudit classifies all nine UD rungs as mislabeled, because each carries the same 194 forced tensors. We split that class in two, because the consequence differs — for UD-IQ4_XS the substitution runs IQ4_XS to IQ4_NL, 4.25 to 4.50, and the file’s average does not move, so the label still predicts the size. Read ⚠ as “flagged by the tool, harmless in practice.”

The interesting row is the first one, and the number everyone quotes needs a sentence of context. The verdict line for UD-IQ1_S reads “51.9% of parameters cannot carry the labeled quant family.” That figure is exact — it is the share of parameters the label could not apply to. It is not a damage figure, and the breakdown shows why:

  nominal: IQ1_S -> base type IQ1_S = 1.562 bpw     actual overall: 3.279 bpw   (+110% vs nominal)

  routed experts             144    68.3%    54.9%    2.64   IQ1_S(68) IQ4_NL(48) IQ2_XXS(28)

  DIMENSION-FORCED SUBSTITUTIONS: 194 tensors
    other                     1 tensors -> IQ4_NL  (4.50 bpw)  ncols=160 (%256=160)   28.9% of params
    routed experts           48 tensors -> IQ4_NL  (4.50 bpw)  ncols=640 (%256=128)   22.8% of params

More than half of the forced share is a single tensor: the model’s per-layer n-gram embedding table, 160 wide, holding 28.9% of all parameters. Flash-Next’s architecture puts a 51-billion-parameter lookup table beside a 125-billion-parameter MoE, and Unsloth’s documentation says, in so many words, that this table is kept at 4 bits on purpose: “these are not quantized that heavily (4-bit minimum) since they have random access pattern, and quantizing them heavily will damage the model.” So the fallback moved that table from a requested 4.25 to an actual 4.5 bits per weight. That is a real substitution, correctly detected, and it is not damage of any kind.

The share that does mean something is the second line: 48 routed-expert tensors, 640 wide, 22.8% of the model, requested at 1.56 bits and written at 4.5. The other 96 expert tensors in the same file — the ones with widths that divide by 256 — did get their 1- and 2-bit types, 68 of them at a genuine IQ1_S. So the file is a real low-bit quantization of about two thirds of its experts, a 4.5-bit copy of the other third, and a 4.5-bit lookup table that was always going to be there. That is why it is 72.5 GB and why the measured average is 3.28.

Credit for that reading belongs to the thread, not to us. The original post said “51.9% of parameters forced into fallback types” without the breakdown; a commenter pointed at Unsloth’s 4-bit-minimum note; and about three hours later the author had rerun the numbers and written “The real forced part is the experts, about 23%, not 52%. Census updated. File’s still 3.28 bpw under a 1.56 label, but I was blaming the wrong tensor.” The census now carries that footnote. It is a good example of what a public audit should do when it is corrected, and it is also a warning for anyone quoting the tool: the percentage measures how much of the file the label cannot describe, not how much of the file was hurt. Those come apart whenever a maker was holding a tensor at high precision anyway.

For a control, we ran the same command against a dense model from the same maker, unsloth/Qwen3.8-27B-GGUF, on its UD-IQ2_XXS file. Label 2.06, measured 2.16, no forced tensors, 355 tensors whose type differs from the label — all of them the dynamic recipe’s deliberate choices on widths that divide by 256 — and a verdict of “label is honest.” The +5% is what a good mixed recipe costs. The +110% is what a width of 640 costs.

What the label cannot tell you — and what the tool cannot either

Let me say what has and has not been shown. The tool proves a narrow thing: on these tensors, the labelled type is mathematically impossible, and a fallback-shaped type sits there instead. It does not re-derive what type the recipe would have chosen for each tensor — that is hundreds of lines of architecture-specific logic in llama_tensor_get_type — and it cannot distinguish “the quantizer forced this to Q8_0” from “the maker chose Q8_0 here and the width happened not to divide.” Its README says so. For the purpose of knowing whether a file fits in memory, the distinction does not matter: the bytes are the bytes. For the purpose of assigning fault, it matters a great deal, and the answer is nobody.

Nor does a higher measured density mean a worse file. It means more bits, which on the evidence of our FP8 post is usually good news for output quality. Unsloth’s own table for Flash-Next has UD-IQ1_S retaining 80.2% top-1 agreement with BF16 and UD-IQ3_XXS retaining 87.6% (their figures, their benchmark) — a gap that is real and that the fallback did not erase. The point is only what the FAQ of the tool puts more sharply than I would: a low-bit quantization exists to fit a memory budget, and “a file labeled 2.06 bpw that is actually 4.58 bpw fails at its only job: telling you whether it fits.”

There is also a hardware note the label cannot carry. One of this week’s field reports from a 128 GB M4 Max says its author picked UD-Q3_K_XL because 90 GB looked like it would fit, and only afterwards learned that IQ-family types decode through codebook lookups Apple Silicon handles slowly — so a smaller IQ file can run slower than a larger K-quant. Another variable, another thing the filename does not say.

Three steps to read a GGUF’s real bits per weight before you download

The per-tensor types are in the header of every GGUF file, so this is a solved problem for anyone who looks. Three ways to look, from least to most effort:

1. On Hugging Face, click the file. The viewer shows the tensor table with a type per tensor. If a file called IQ2_XXS lists its ffn_*_exps tensors as IQ4_NL, you have your answer without downloading anything.
2. Locally, gguf-dump. llama.cpp’s gguf-py package prints metadata and the tensor table without loading weights; 32-block types on the big tensors are the tell.
3. Remotely, one command. python3 ggufaudit.py hf://owner/repo --quant <label> reads the header over the network and prints nominal against actual overall, then a DIMENSION-FORCED SUBSTITUTIONS block. Read those two lines and stop. The label has to match the filename as published — Unsloth’s rungs carry a UD- prefix, so it is --quant UD-IQ1_S, not --quant IQ1_S; a label that matches nothing prints a usage message rather than an error.

The rule for what to do with the result is short enough for a table.

And one practical consequence that the census author put well: on a fallback-dominated model, the rungs at the bottom of the ladder may not be different products. On Nemotron-3.5-Lightning, IQ2_XXS and IQ2_M are the same density and about 21 MB apart; choosing between them by filename is choosing between two names for one file. On Flash-Next the four smallest files span 9.5 GB and 0.43 bits, a real but small difference, bought by pushing the divisible two-thirds of the experts from 3-bit to 1-bit while the other third stays at 4.5 regardless. If your memory budget is the reason you are at that end of the ladder, the label is not the number to budget with.

Back to the routing table: a declared quantization is also a label

The earlier posts in this series ended at a routing table with four fields, and the advice was to route on what the host declares and to rank unknown below any declared value. I still think that is right. What this week adds is a qualification I should have made then: a declared value is a claim made in good faith by someone reading a label, and the label can be honestly wrong. The GGUF case is the cleanest demonstration I know of — every party did the standard thing, and the file still is not what it says. Nothing in the API-serving world is different in kind. When a host reports fp8 in an endpoint listing, that is a declaration, not a measurement, and the only way to turn it into a measurement is the one both earlier posts landed on: run your own prompts against a reference you trust and look at the divergence.

So the ordering has three levels rather than two: measured beats declared, declared beats unknown, and the person holding the memory budget is the only one in a position to do the measuring. For a GGUF file that takes a few megabytes of header reads and one command. Run it before the download — a 72.5 GB file will tell you what it is eventually, but only one of those orders leaves you the disk space to do something about it.