Tagged “quantization”
-
Working Within 8 GB of VRAM
8 GB is enough for real local LLM work if you spend it deliberately. How to budget weights against context, and which trade-offs to make first.
-
Understanding GGUF and Quantization Levels
What GGUF is, how to read quantization labels like Q4_K_M, and how to choose a quant that balances quality and memory for your hardware.
-
How Much VRAM and RAM Do You Actually Need to Run an LLM?
A practical guide to sizing local LLMs: how to estimate VRAM and RAM from parameters and quantization, plus what happens when a model doesn't fit.
-
Is That Quantization Good Enough? Test It, Don't Guess
Stepping down a quant level to fit your GPU is usually worth it — but the only way to know for your work is a small repeatable check you run yourself.
-
Running a GGUF Your Runtime's Library Doesn't Offer
Curated model libraries are a subset. How to find, vet, and load an arbitrary GGUF file yourself when the quant or model you want isn't listed.