Tagged “vram”
-
Why Your Local Model Is Slow: A Diagnostic Order
Local inference feels sluggish for a handful of reasons. Check them in this order — memory spill first — and you'll find the cause without guessing.
-
What to Look For in a GPU for Local Inference
The GPU properties that actually govern local LLM performance — capacity first, then bandwidth — and why the spec people shop on matters least.
-
Working Within 8 GB of VRAM
8 GB is enough for real local LLM work if you spend it deliberately. How to budget weights against context, and which trade-offs to make first.
-
Local LLMs on Apple Silicon: What Unified Memory Changes
Why Macs punch above their weight for local inference, where the bandwidth ceiling bites, and how to think about memory when CPU and GPU share a pool.
-
GPU Layer Offload, and Why Partial Is the Worst Place to Be
How local runtimes split a model between GPU and CPU, why a few spilled layers cost so much speed, and how to decide what to cut instead.
-
What a Long Context Window Costs on Local Hardware
Context isn't free locally — it takes memory for the KV cache and time on every turn. How to size a context window to what you actually use.
-
How Much VRAM and RAM Do You Actually Need to Run an LLM?
A practical guide to sizing local LLMs: how to estimate VRAM and RAM from parameters and quantization, plus what happens when a model doesn't fit.
-
Prompt Processing vs Token Generation: Two Different Speeds
Local inference has two phases with opposite bottlenecks. Knowing which one you're waiting on tells you whether to change the model or the prompt.
-
What Happens When Two Requests Hit Your Local Model at Once
Local runtimes handle concurrency very differently from hosted APIs. What queueing and batching cost you, and how to plan for more than one caller.