How Much VRAM and RAM Do You Actually Need to Run an LLM?

The single question that decides whether a local model runs well is simple: does it fit in memory? Get this right and inference is fast and smooth. Get it wrong and the model either refuses to load or crawls along at a word every few seconds. The good news is you can estimate the answer with arithmetic you can do in your head.

These are rules of thumb, not guarantees — real usage depends on the runtime, context length, and your exact hardware. Use them to get in the right ballpark, then verify.

Start from the file size

The most reliable estimate isn’t from the parameter count directly — it’s from the size of the model file you’re about to download. A quantized model on disk is close to how much memory the weights will occupy when loaded. So a 4.7 GB model file needs roughly 4.7 GB just to hold the weights, plus overhead.

A working rule: you want VRAM (or RAM) somewhat larger than the model file — the weights have to fit, plus headroom for the context window and the KV cache (more on that below). Budgeting 20–30% above the file size for a normal context length is a reasonable starting point.

Estimate from parameters and quantization

If you only know the parameter count, you can estimate the file size from the quantization. The key number is bits per parameter:

FP16 (full precision as usually shipped): 16 bits = 2 bytes per parameter.
8-bit quantization: ~1 byte per parameter.
4-bit quantization: ~0.5 byte per parameter, plus a little metadata.

So the memory for the weights is roughly parameters × bytes-per-parameter:

A 7B model at FP16: 7B × 2 ≈ 14 GB.
The same 7B at 8-bit: ≈ 7 GB.
The same 7B at 4-bit: ≈ 4 GB.

That’s why quantization is the whole game for local inference. Dropping from 16-bit to 4-bit shrinks a model by roughly 4×, turning a model that needs a data-center GPU into one that fits on a laptop — with only a modest quality cost at the common 4-bit levels.

Rough weight sizes at 4-bit for popular sizes: 7–8B ≈ 4–5 GB, 13–14B ≈ 8–9 GB, 30–34B ≈ 18–20 GB, 70B ≈ 40 GB. Add headroom on top of each.

Don’t forget the KV cache and context

The weights are the big cost, but not the only one. As the model processes a conversation, it keeps a KV cache — intermediate state for every token in the context window. This grows with context length: a short prompt costs almost nothing, but filling a large context window (tens of thousands of tokens) can add anywhere from a few hundred megabytes to several gigabytes depending on the model. If you plan to use long contexts, leave more headroom than the file size alone suggests.

VRAM vs system RAM vs unified memory

Where that memory lives matters as much as how much you have:

GPU VRAM is by far the fastest place to run a model. If the whole model fits in VRAM, you get the best tokens-per-second. This is the target.
System RAM with CPU inference works and is a great fallback, but it’s much slower — fine for smaller models and casual use, painful for large ones.
Apple Silicon unified memory blurs the line: the CPU and GPU share one fast memory pool, so a Mac with, say, 32 GB of unified memory can run models that would need a 32 GB GPU on a PC. This is why Macs punch above their weight for local LLMs.

A practical target: match the model’s memory need to your fastest sufficient pool. An 8 GB GPU comfortably runs 7–8B models at 4-bit. A 24 GB GPU opens up 30B-class models. Beyond that you’re into multi-GPU or heavy system-RAM territory.

What happens when it doesn’t fit

Understanding the failure modes helps you diagnose a slow setup:

Partial offload. Many runtimes (llama.cpp and tools built on it, like Ollama) can put some layers on the GPU and the rest on the CPU. The model runs, but speed drops toward CPU levels the more layers spill over. If a model is “working but slow,” this is usually why.
Won’t load at all. If even partial loading exceeds available memory, the runtime errors out or the process is killed.
Thrashing. On an overcommitted machine the OS may swap to disk, which makes inference effectively unusable.

The fix is always the same lever: use a smaller model, a smaller quantization, or a shorter context — whichever costs you the least for your use case.

A quick sizing checklist

Find the model file size (or estimate: parameters × 0.5 byte for 4-bit).
Add ~25% for overhead, more if you want long contexts.
Compare against your fastest memory pool — GPU VRAM first, then unified memory, then system RAM.
If it doesn’t fit comfortably, step down a quant level or a model size before anything else.

Sizing is the one bit of local-LLM planning worth doing before you download 40 GB of weights. Once you can eyeball it, choosing a model that will actually run well becomes second nature. Because runtimes and quant formats keep evolving, confirm the specifics for your tool against its current documentation.