What Happens When Two Requests Hit Your Local Model at Once

A hosted API absorbs concurrent requests invisibly because there’s a fleet behind it. Your local runtime has one copy of the weights and one GPU, so two simultaneous requests have to either take turns or share — and which one your runtime does, and how well, decides whether a multi-user tool is viable at all on your hardware.

The two strategies

Serialisation. Requests queue and run one at a time. Simple, predictable, and it means the second caller waits for the first to finish completely. Throughput equals single-request throughput; latency for anyone but the first caller is terrible.

Continuous batching. The runtime processes several sequences together, generating a token for each in the same pass. This is a genuine win, and the reason is worth understanding: generation is memory-bandwidth-bound, and streaming the weights once to produce tokens for several sequences amortises the expensive part. Total throughput rises substantially while per-request speed drops only somewhat.

Support varies. Purpose-built serving stacks are built around batching; general-purpose local runtimes range from serialising to batching with a configurable limit, and it has changed over versions. Check your runtime’s current documentation rather than assuming — this is exactly the kind of behaviour that gets improved quietly.

The memory cost nobody plans for

Batching is not free. Each concurrent sequence needs its own KV cache, and the cache is per-token-per-sequence. So the memory arithmetic changes: instead of weights plus one context’s worth of cache, you need weights plus one cache per concurrent slot.

This is where local concurrency usually breaks. A model that fits comfortably for one user at a long context may not fit for four, and the failure mode is the familiar one — layers spill to the CPU and everything gets slow, per GPU layer offload explained.

Two ways to make it fit:

• Reduce context per slot. Concurrency and context length trade directly against each other in the same budget. Four users at a modest context may work where one user at a huge context does — see what a long context window costs.
• Use a smaller model. Frees room for more slots. For a shared setup, a small model serving everyone quickly usually beats a large model serving one person while others wait.

Deciding how much concurrency you need

Be honest about the actual pattern, because the answer is often “one.”

Genuinely single-user. You, one chat window or one editor plugin at a time. Serialisation is fine and you should optimise for single-request latency instead.

One user, several tools. More common than people notice: an editor assistant, a chat front-end, and a script all pointed at the same endpoint. They rarely fire simultaneously, but when they do you’ll get a stall. Modest concurrency, or accepting occasional queueing, is usually enough.

Batch work. A script processing many items has natural concurrency and it’s the case that benefits most from batching — you care about total throughput, not per-item latency. Sending a handful of requests in flight rather than strictly one at a time can improve total time substantially if your runtime batches.

Multiple people. A shared model for a household or small team is where this stops being academic. Expect to size for concurrent slots explicitly, keep context modest, and consider whether a general-purpose local runtime is the right tool — serving stacks designed for throughput exist for this reason.

Practical guidance

Test with your real concurrency before you rely on it. Fire the number of simultaneous requests you expect and watch both latency and whether the model stayed on the GPU. Behaviour under load is not predictable from single-request behaviour.

Set a concurrency limit deliberately. Unbounded concurrency against a fixed memory budget is how you get the worst outcome: everything spills, and every caller is slow. A queue in front of a healthy model is better than no queue in front of a thrashing one.

Enable streaming for anything a human watches. Under batching, per-request generation is slower, and streaming hides much of that — see serving a local model over HTTP.

Set generous client timeouts. A request that queued behind two others can take a long time. Add cold starts on top and short timeouts will fail for reasons that look nothing like the cause — model load time and keeping it warm.

Don’t run two models to get concurrency. Two resident models double the weight memory and halve what’s available for cache. One model with several slots is nearly always the better use of a fixed budget.

When to stop trying

There’s a point where the honest answer is that your hardware isn’t a multi-user inference server. If several people need low-latency access to a capable model, one consumer GPU will disappoint all of them, and the trade-offs in local vs cloud shift accordingly. A good middle path: keep the private, high-volume, latency-tolerant work local and send the interactive burst load elsewhere.

A practical takeaway

Budget memory per concurrent slot, not per model. Then test at your real concurrency and check the model is still fully on the GPU — that one check catches most of what goes wrong when a local endpoint acquires a second caller.