Understanding GGUF and Quantization Levels

If you’ve browsed local models, you’ve seen filenames like model-Q4_K_M.gguf and wondered what all of it means. Two ideas are tangled together there: GGUF, a file format, and quantization, the compression that makes local inference practical. Once you can read these labels, picking the right download stops being guesswork.

Quant naming conventions have shifted over time and vary a little between model publishers, so treat the specific labels here as the common convention and check the model card for its own notes.

What GGUF is

GGUF (GGML Universal Format) is the file format used by llama.cpp and the many tools built on it — including Ollama and LM Studio. It’s a single self-contained file that packages the model weights along with the metadata a runtime needs: the architecture, tokenizer, and default parameters. That self-containment is the point — you download one .gguf file and any GGUF-capable runtime can load it without extra config.

GGUF replaced the older GGML format, which lacked this metadata and broke compatibility whenever the format changed. If you run into references to .ggml files or “GGML models,” that’s the previous generation; GGUF is what current tooling uses.

Crucially, GGUF is designed for CPU and mixed CPU/GPU inference. It’s why you can run a capable model on a laptop with no discrete GPU, and why llama.cpp-based tools can offload just some layers to a GPU. Other formats exist for pure-GPU stacks (like the safetensors weights used with GPU-only runtimes), but GGUF is the workhorse of local, hardware-flexible inference.

What quantization actually does

A model’s weights are originally trained as high-precision numbers, usually 16-bit floats. Quantization stores them at lower precision — 8-bit, 5-bit, 4-bit, or even less — so each weight takes far fewer bits. The result is a dramatically smaller file that needs less memory and often runs faster, because there’s less data to move around.

The trade-off is quality. Rounding each weight to fewer bits introduces small errors. At mild quantization the loss is barely measurable; push too far and the model gets noticeably worse — repetitive, less accurate, sometimes incoherent. The art is choosing the most aggressive quantization that still gives output you’re happy with, so you can run the biggest useful model your hardware allows.

Reading a quant label

Take Q4_K_M apart:

Q4 — the nominal bit width. Q4 is roughly 4 bits per weight, Q5 roughly 5, Q8 roughly 8. Lower = smaller and faster, but lower quality.
_K — a k-quant, the modern quantization scheme that allocates bits more cleverly across a model than the older “legacy” quants (which you’ll sometimes see as Q4_0 or Q4_1). K-quants generally give better quality at the same size.
_M — the size variant within that scheme: S (small), M (medium), L (large). A larger variant spends a few more bits on the most important weights for a little more quality and size.

So Q4_K_M reads as “4-bit k-quant, medium variant” — and it’s the most commonly recommended default because it sits at a sweet spot of quality per gigabyte.

Choosing a quant level

A practical way to think about the ladder, from most to least compressed:

Q2 / Q3 — smallest, fastest, but quality drops off, sometimes sharply. Use only when memory is tight and you’ve confirmed the output is still good enough. Larger models tolerate low bit-widths better than small ones.
Q4 (especially Q4_K_M) — the default recommendation for most people. Big size savings, quality loss that’s usually minor and hard to notice in everyday use.
Q5 (Q5_K_M) — a step up in fidelity for a bit more memory. A good choice if you have the headroom and want to be safe.
Q6 / Q8 — close to the original quality, with 8-bit being nearly indistinguishable from full precision for most tasks. Use when you have plenty of memory and quality matters more than speed.

A useful heuristic: prefer a larger model at a lower quant over a smaller model at a higher quant, when both fit. A 13B model at Q4 usually beats a 7B model at Q8, because raw capability from more parameters tends to matter more than the last bit of precision. Test on your own tasks — the right answer depends on what you’re asking the model to do.

Putting it together

When you go to download a model, the workflow is:

Estimate your memory budget (see our VRAM/RAM sizing guide) — how large a file can you comfortably fit?
Pick the largest model whose quantized file fits that budget.
Choose the quant for that model — start at Q4_K_M, move up to Q5/Q6 if you have room and want more fidelity, or down to Q3 only if you must.
Try it on real prompts and step up or down based on whether the quality holds.

Tools like Ollama hide most of this by shipping a sensible default quant, which is fine to start. But the moment you want to squeeze a bigger model onto your hardware, or you notice output quality dipping, knowing how to read Q4_K_M lets you make the call deliberately instead of hoping. As always with fast-moving tooling, confirm current quant options and recommendations on the model’s own page before committing to a large download.