Prompt Processing vs Token Generation: Two Different Speeds

Local inference happens in two phases with genuinely different performance characteristics. Reading in your prompt is parallel and compute-bound; producing the answer is sequential and memory-bandwidth-bound. They respond to different fixes, and most confusion about local LLM speed comes from treating them as one number.

Phase one: reading the prompt

Before any output appears, the model processes your entire input. Crucially, it can do this for all input tokens at once — they’re all already known, so the work parallelises well and keeps the hardware’s arithmetic units busy.

Properties worth knowing:

It scales with input length. Twice the prompt, roughly twice the wait (somewhat more, in fact, since attention cost grows faster than linearly).
It’s compute-bound. This is the phase where raw GPU throughput helps, and where a powerful chip earns its keep.
It’s a one-time cost per prompt — but in a chat, “per prompt” means the whole conversation history each turn unless the runtime caches it.

The user-visible symptom is a pause before the first token, sometimes called time-to-first-token.

Phase two: producing the answer

Generation is unavoidably sequential. Each token depends on the one before it, so there’s nothing to parallelise across the output. And producing one token requires reading essentially the whole weight set out of memory while doing comparatively little arithmetic with each value.

That combination makes generation memory-bandwidth-bound: the hardware spends its time waiting on memory rather than computing. Which explains a set of otherwise-puzzling observations:

A smaller model generates faster, roughly in proportion to its size — fewer bytes to stream per token.
A smaller quantization also generates faster, for the same reason. The speed gain from dropping a quant level is not about easier maths.
Memory bandwidth predicts generation speed better than compute does when comparing hardware. See what to look for in a GPU.
A partially-offloaded model collapses, because part of the weight stream now comes from system RAM over a much slower path — GPU layer offload explained.

Telling them apart, and what to do

The diagnosis is easy once you know to make it. Ask: is the wait before text appears, or is the text appearing slowly?

Long pause, then reasonable flow. Prompt processing dominates. Your input is the cost, not the model’s size. Useful responses:

Send less. Extract the relevant section rather than the whole document — usually improves answer quality too.
Check whether your runtime caches the processed prompt between requests. Many can reuse the work for a shared prefix, which turns a repeated system prompt or an ongoing conversation from a recurring cost into a one-time one. Support and configuration vary; check current documentation.
In a long chat, start fresh with a summary rather than carrying the whole history — what a long context window costs.
A step down in model size helps less here than you’d hope, since this phase is compute-bound.

Text appears immediately but crawls. Generation is the cost. Responses:

Step down a model size or a quantization level — direct, proportional wins.
Verify the whole model is on the GPU. If it isn’t, that’s your answer and nothing else matters until it’s fixed.
Ask for shorter output. Verbose models are expensive locally in a way they aren’t in the cloud, and a prompt that specifies brevity genuinely saves time.
Nothing about your prompt length will help.

Both are slow. Usually one underlying cause — memory pressure or contention — rather than two problems. Work through the diagnostic order.

Why this shapes what local models are good at

Put the two phases together and a pattern emerges: local inference is relatively good at long-input, short-output work and relatively poor at short-input, long-output work. Summarising a document, extracting fields, classifying text, answering a question about a file — all of these read a lot and write a little, and the expensive phase is the one that parallelises.

Generating a long article from a one-line prompt is the opposite shape, and it’s where waiting for a local model feels worst. Reasoning-tuned models that produce long visible working are also expensive for this reason: all that intermediate text is generated one token at a time.

This is worth knowing when choosing what to run locally versus what to send to a hosted model — the shape of the request matters as much as the difficulty. It’s a useful refinement to the broader local vs cloud trade-offs.

A practical takeaway

Time the two phases separately before you tune anything. Prompt-bound means change the prompt; generation-bound means change the model or the quantization. Comparing a single wall-clock number between configurations mixes both signals and will send you optimising the wrong half.