Is That Quantization Good Enough? Test It, Don't Guess

Almost every local-LLM decision eventually becomes “will a smaller quantization hurt?” The honest answer is that it depends on the model, the level, and your task, and no published number will settle it for the work you actually do. What settles it is a twenty-minute check you can repeat whenever the question comes back.

If you’re still getting oriented on what the labels mean, start with understanding GGUF and quantization levels; this post is about deciding between two of them.

Why general guidance only gets you so far

The broad shape is well established: quantization loses some fidelity, the loss is small at moderate levels and grows as you get more aggressive, and lower-precision variants of a larger model often beat higher-precision variants of a smaller one. That’s a genuinely useful prior.

What it can’t tell you is where your threshold sits, because the effect isn’t uniform:

Task sensitivity varies a lot. Casual chat and rough drafting tolerate degradation well. Strict format adherence, code that has to parse, and precise extraction expose it much faster.
Models differ. Two models of the same size don’t degrade identically at the same level.
Failure is qualitative, not a percentage. What you’ll notice isn’t “5% worse” — it’s the model drifting from an instructed format, repeating itself, or losing the thread on longer inputs.

That last point is why an aggregate quality score wouldn’t help even if you had one. You care about specific failures in specific work.

The check

The goal is a comparison you can run in an evening and re-run in ten minutes later.

1. Build a fixed prompt set. Ten to twenty prompts from your real work, saved in a file so they’re identical every time. Include your hardest cases deliberately: the long input, the strict output format, the ambiguous request. Easy prompts pass at every quant level and tell you nothing.

2. Fix everything except the quantization. Same model family and size, same context length, same sampling settings, same prompts. If you also change the model, you’ve learned nothing about quantization.

3. Set sampling to be as deterministic as your runtime allows. Random sampling will produce different answers from the same model on the same prompt, which drowns out the difference you’re trying to measure. Reducing temperature toward zero — and fixing a seed if your runtime supports it — makes the comparison legible. You can go back to your usual settings afterwards.

4. Run both and save the output. Actually write it to files. Reading two answers side-by-side is far more informative than remembering how the last one felt.

5. Judge on failures, not on style. Ask, per prompt: did it follow the instruction? Is the format valid? Are the facts drawn from the input correct? Did it stop when it should? Style differences between quant levels are noise; broken instruction-following is signal.

What degradation actually looks like

Knowing the symptoms makes them much easier to spot:

Format drift. You asked for JSON, or a table, or three bullets, and it mostly complied but not exactly. Usually the first thing to go, and the most consequential if a program is parsing the output.
Repetition and looping. Repeating a phrase, or failing to stop. A strong sign you’ve gone a level too far.
Losing distant context. The answer is fine about the start of a long input and drifts on the end.
Confident small errors. Names, numbers, and identifiers from the input subtly wrong. The most dangerous mode because the output still reads well.
Weaker multi-step work. Anything requiring several dependent inferences degrades before single-step recall does.

If none of these appear across your hard prompts, the smaller quant is fine for your work. Ship it and take the memory back.

What you gain, so you can weigh it

Be explicit about the other side of the trade. Stepping down a level buys you:

Memory, which may be what gets your whole model onto the GPU. That’s not a small improvement, it’s the difference between two performance regimes — GPU layer offload explained.
Generation speed, roughly in proportion to the size reduction, because generation is bandwidth-bound.
Context length, if you spend the freed memory on the KV cache instead.

When a step down closes a partial-offload gap, it’s almost always worth it. When you already fit comfortably on the GPU with room for context, there’s little reason to go smaller and you may as well keep the fidelity.

A practical takeaway

Keep the prompt set. It’s the asset — it makes this question answerable in minutes rather than an evening, and it does double duty when you’re picking a model for a job or checking that an update didn’t change behaviour. The quantization question recurs every time your hardware or your models change; the test shouldn’t have to be rebuilt each time.