Tagged “performance”
-
Feeding Web Pages to a Local Model Without Wrecking the Context
Raw HTML is mostly not content, and locally every token costs VRAM and wall-clock time. Why markdown is the format a small model can actually read.
-
Why Your Local Model Is Slow: A Diagnostic Order
Local inference feels sluggish for a handful of reasons. Check them in this order — memory spill first — and you'll find the cause without guessing.
-
Serving a Local Model Over HTTP to Your Own Apps
How local LLM runtimes expose an HTTP endpoint, why the OpenAI-compatible shape matters, and how to wire your own scripts and tools into a local model.
-
What to Look For in a GPU for Local Inference
The GPU properties that actually govern local LLM performance — capacity first, then bandwidth — and why the spec people shop on matters least.
-
Working Within 8 GB of VRAM
8 GB is enough for real local LLM work if you spend it deliberately. How to budget weights against context, and which trade-offs to make first.
-
Local vs Cloud LLMs: Privacy, Cost, and When Each Wins
An honest comparison of running LLMs locally versus using cloud APIs — privacy, cost, latency, capability, and which to choose for real use cases.
-
Local LLMs on Apple Silicon: What Unified Memory Changes
Why Macs punch above their weight for local inference, where the bandwidth ceiling bites, and how to think about memory when CPU and GPU share a pool.
-
GPU Layer Offload, and Why Partial Is the Worst Place to Be
How local runtimes split a model between GPU and CPU, why a few spilled layers cost so much speed, and how to decide what to cut instead.
-
What a Long Context Window Costs on Local Hardware
Context isn't free locally — it takes memory for the KV cache and time on every turn. How to size a context window to what you actually use.
-
Prompt Processing vs Token Generation: Two Different Speeds
Local inference has two phases with opposite bottlenecks. Knowing which one you're waiting on tells you whether to change the model or the prompt.
-
CPU-Only Inference: When It's Genuinely Enough
No GPU is not the same as no local LLM. Where CPU inference works well, what governs its speed, and which workloads suit it best.
-
Is That Quantization Good Enough? Test It, Don't Guess
Stepping down a quant level to fit your GPU is usually worth it — but the only way to know for your work is a small repeatable check you run yourself.
-
Model Load Time, Cold Starts, and Keeping a Model Warm
The first request after an idle period pays to read gigabytes from disk. Why that happens, when to prevent it, and what keeping a model resident costs.
-
What Happens When Two Requests Hit Your Local Model at Once
Local runtimes handle concurrency very differently from hosted APIs. What queueing and batching cost you, and how to plan for more than one caller.
-
Updating Models and Runtimes Without Breaking What Works
Local LLM tooling moves fast, and updates change behaviour. How to upgrade deliberately, verify nothing regressed, and keep a way back.