Why Your Local Model Is Slow: A Diagnostic Order
A local model that’s slow is almost never slow for a mysterious reason. There’s a short list of causes, they have very different fixes, and they’re distinguishable if you check them in the right order. The single most common cause by a wide margin is that the model doesn’t entirely fit where you think it does.
Work down this list rather than changing settings at random. Each step tells you whether to stop or continue.
1. Is the whole model on the GPU?
Start here, always. When a model doesn’t fit in VRAM, most runtimes don’t fail — they put some layers on the GPU and run the rest on the CPU. It works. It’s just far slower, and the slowdown scales with how much spilled over.
The tell is a mismatch between what you expected and what you got, with no errors anywhere. Your runtime will usually report how many layers were placed on the GPU at load time; that line is the most valuable diagnostic output you have.
If layers are spilling, you have three levers and they all trade something: a smaller quantization, a smaller model, or a shorter context. See GPU layer offload explained for how to reason about the split, and don’t tune anything else until this is resolved. No sampling setting will recover what a CPU fallback costs you.
2. Are you measuring the prompt or the generation?
These are two different operations with different performance characteristics, and confusing them sends people down the wrong path. Reading in a long prompt is one kind of work; producing tokens one at a time is another, and it is the one bounded by memory bandwidth.
The practical symptom: a long pause before any output appears, then reasonable speed once text starts flowing, means your prompt processing is the cost — not the model’s generation speed. That’s a context-size problem, not a model-size problem. Prompt processing vs token generation covers why the two scale so differently and what to do about each.
3. Is this actually a cold start?
If the slowness is only on the first request after a while, you’re measuring model loading, not inference. Weights have to be read from disk into memory, and for a multi-gigabyte file on a slow disk that is a real wait.
Ask the same question twice in a row. If the second answer is fast, this is your cause, and the fix is about keeping the model resident rather than about performance tuning — model load time and keeping it warm.
4. Is the context window doing damage?
A large context costs memory for the KV cache, and that memory competes with the weights for the same VRAM. Setting a very large context “just in case” can push you back into case 1 without you connecting the two changes.
It also costs time directly: a conversation that keeps growing re-sends its whole history, so a chat that started fast gets slower turn by turn. If your slowdown developed over a long session, that’s this. What a long context window costs goes through the arithmetic.
5. Is something else contending for the hardware?
Boring but common. A browser with hardware acceleration, a game, another model still resident, a compile job — all of them take VRAM or memory bandwidth. On a machine with unified memory, an application holding a lot of RAM directly reduces what inference can use.
Check what else is loaded before assuming the model is the problem. Two models left resident from earlier experiments is a classic.
6. Is the machine thermally limited?
Sustained inference is a sustained load. Laptops in particular will run fast for a short burst and then settle to a lower clock. If a long generation starts quick and degrades, or performance is worse than yesterday for no reason you changed, look at cooling and power settings — including whether the machine is on battery, since many will cap GPU power aggressively when unplugged.
7. Only now, look at settings
Thread counts, batch sizes, and backend choices are real levers, but they produce modest gains compared to the earlier items and they’re easy to make worse. In particular, more threads than you have physical cores usually hurts. Change one thing, re-test with the same prompt, and keep a note of what you tried.
The shape of the answer
Almost every “my local model is unusably slow” story resolves to case 1: something didn’t fit, so it silently went to the CPU. The second most common is case 3, someone timing a cold start once. Both are diagnosable in a couple of minutes if you know to look, which is the whole point of having an order.
When you do want to compare, always compare with the same prompt at the same context length. Local inference speed varies enough between runs and prompt shapes that casual before-and-after impressions will mislead you.