Tagged “hardware”
-
How to Pick a Local Model for the Job You Have
A repeatable way to choose an open-weight model for a specific task — filter by what fits, shortlist by design intent, then decide on your own work.
-
Why Your Local Model Is Slow: A Diagnostic Order
Local inference feels sluggish for a handful of reasons. Check them in this order — memory spill first — and you'll find the cause without guessing.
-
What to Look For in a GPU for Local Inference
The GPU properties that actually govern local LLM performance — capacity first, then bandwidth — and why the spec people shop on matters least.
-
Working Within 8 GB of VRAM
8 GB is enough for real local LLM work if you spend it deliberately. How to budget weights against context, and which trade-offs to make first.
-
Local vs Cloud LLMs: Privacy, Cost, and When Each Wins
An honest comparison of running LLMs locally versus using cloud APIs — privacy, cost, latency, capability, and which to choose for real use cases.
-
Local LLMs on Apple Silicon: What Unified Memory Changes
Why Macs punch above their weight for local inference, where the bandwidth ceiling bites, and how to think about memory when CPU and GPU share a pool.
-
Understanding GGUF and Quantization Levels
What GGUF is, how to read quantization labels like Q4_K_M, and how to choose a quant that balances quality and memory for your hardware.
-
GPU Layer Offload, and Why Partial Is the Worst Place to Be
How local runtimes split a model between GPU and CPU, why a few spilled layers cost so much speed, and how to decide what to cut instead.
-
What a Long Context Window Costs on Local Hardware
Context isn't free locally — it takes memory for the KV cache and time on every turn. How to size a context window to what you actually use.
-
How Much VRAM and RAM Do You Actually Need to Run an LLM?
A practical guide to sizing local LLMs: how to estimate VRAM and RAM from parameters and quantization, plus what happens when a model doesn't fit.
-
Prompt Processing vs Token Generation: Two Different Speeds
Local inference has two phases with opposite bottlenecks. Knowing which one you're waiting on tells you whether to change the model or the prompt.
-
CPU-Only Inference: When It's Genuinely Enough
No GPU is not the same as no local LLM. Where CPU inference works well, what governs its speed, and which workloads suit it best.
-
Model Load Time, Cold Starts, and Keeping a Model Warm
The first request after an idle period pays to read gigabytes from disk. Why that happens, when to prevent it, and what keeping a model resident costs.
-
What Happens When Two Requests Hit Your Local Model at Once
Local runtimes handle concurrency very differently from hosted APIs. What queueing and batching cost you, and how to plan for more than one caller.
-
Running a Local LLM in a Container: What's Worth the Trouble
Containers make a local model reproducible and portable — but GPU passthrough and model storage are the two things that decide whether it's worth it.