GPU Layer Offload, and Why Partial Is the Worst Place to Be
When a model is too large for your GPU, most local runtimes don’t refuse to run it. They put as many layers as fit on the GPU and execute the rest on the CPU. The model works, no error appears, and it is dramatically slower than you expected. Understanding this mechanism is the single most useful thing you can know about local inference performance.
What a layer split actually means
A transformer model is a stack of largely identical layers, and each token’s computation passes through all of them in order. That structure makes the model easy to divide: put the first N layers on the GPU, keep the remainder in system RAM for the CPU to handle, and run them in sequence.
Runtimes expose this as a count of layers to offload, and often default to fitting as many as they can. Two properties of the arrangement explain everything else:
- Execution is sequential, not parallel. Every token waits for the GPU portion, then the CPU portion. You don’t get the GPU and CPU working simultaneously on the same token.
- Data crosses between them on every token. Intermediate state moves over the system bus each pass, which is far slower than either processor’s own memory.
Why a few spilled layers hurt disproportionately
The intuition people bring is linear: 90% of layers on the GPU should give roughly 90% of the speed. It doesn’t, and the reason is that the CPU portion is enormously slower per layer than the GPU portion, so even a small fraction of layers can dominate total time.
Think of it as a relay where one runner is much slower than the others. Their share of the distance barely matters — they set the pace. This is why “it loaded, so it must be fine” is such a misleading signal, and why the fix is never a sampling setting.
There’s a second-order effect too: the bus transfer per token adds latency that scales with nothing you can tune. Partial offload is genuinely the worst regime — slower than you’d expect from the layer ratio, and without the predictability of pure CPU inference.
Diagnosing it
The runtime almost always tells you, at load time, how many layers it placed on the GPU. That line is the diagnostic. If it reports fewer than the model’s total, you are in partial offload, whatever else is happening.
Two secondary tells:
- Fast prompt, slow generation, no errors. The classic profile. If you’re unsure whether you’re looking at this or at a context problem, check the order in why your local model is slow.
- It worked yesterday and doesn’t today. Something else is now holding VRAM, so fewer layers fit. Another model left resident is the usual answer.
The levers, and what each costs
If layers are spilling, you have four options. Ranked by how often they’re the right call:
1. Drop a quantization level. Usually the best trade. Moving to a smaller quant shrinks the weights enough to fit the whole model on the GPU, and the quality cost at common levels is modest compared to what CPU fallback does to speed. Whether the specific step hurts your work is testable — see is that quant good enough.
2. Reduce the context length. The KV cache competes with the weights for the same VRAM, so a large context allocation can be what pushed layers off the GPU. If you set context generously “just in case,” this is free capacity. See what a long context window costs.
3. Step down a model size. A smaller model fully on the GPU generally serves you better than a larger one partially off it. Fewer weights to stream also means faster generation independently of the offload question.
4. Free VRAM elsewhere. Unload other models, close GPU-accelerated applications, and on some systems reduce what the desktop itself is holding. Occasionally this alone closes the gap.
What isn’t a lever: thread counts, batch settings, and sampling parameters. They matter at the margins once the model fits. They cannot recover what the CPU portion costs.
When partial offload is the right answer anyway
It’s a legitimate choice in two situations. First, when the alternative is not running the model at all and you need that specific model’s capability — a slow answer beats no answer. Second, when only a couple of layers spill and the model is the largest you can obtain: the penalty is real but you may prefer it to a size step down.
The rule is to make it a decision. Partial offload chosen knowingly, with expectations set, is fine. Partial offload discovered after an hour of tuning settings is the failure mode worth avoiding.
A practical takeaway
Read the layer count at load. If it’s short of the model’s total, stop tuning and start cutting — quantization first, then context, then model size. Getting the whole model onto the GPU is worth more than every other performance adjustment available to you combined, and the sizing arithmetic will tell you in advance whether a download is going to fit before you spend the bandwidth.