Picking a model that fits your machine
A plain rule of thumb for matching model size to the hardware you already have — no spreadsheet required.
The short answer: start from the memory you already have — GPU VRAM, or unified memory on a Mac — not from a leaderboard. A model has to fit before it can be good to you.
What “fits” actually means
Three things share that memory: the model’s weights, the KV cache that grows as a conversation gets longer, and a bit of runtime overhead. A model that’s slightly too big doesn’t refuse to load — it spills into system RAM and keeps working, just ten times slower or more. That’s the failure mode worth watching for: not a crash, a crawl.
Rough sizes for common hardware
Treat these as starting points, not guarantees — the real number depends on quantization and how much context you run at.
| Hardware | Comfortable size | Notes |
|---|---|---|
| 8 GB VRAM / unified | 7–8B | The most common sweet spot |
| 12–16 GB | 13–14B | Or a 7–8B model with long context |
| 24 GB and up | 30B-class | Or a smaller model with context to spare |
| CPU only, <8 GB | 3B and under | Works, but plan for it to be slow |
Quantization is the lever
The same model usually ships in several sizes depending on how aggressively it’s compressed — a process called quantization. A heavier quantization (smaller file) trades a little quality for a smaller footprint; a lighter one keeps more of the original model’s behavior but costs more memory. Most people land on a middle setting and rarely think about it again. More on reading the actual names in the next post.
When in doubt, go smaller
A fast 7B model you actually use beats a 13B model that makes you wait. You can always step up once you know what you need it for — picking a model isn’t a one-time decision.
Varven’s model picker shows you what’s already on your machine, so you can try a size before committing to it. See how model picking works in Varven.
Common questions
Does more VRAM always mean a better answer?
Not directly. More memory lets you run a larger model or a longer conversation, but a small model with a well-scoped task can outperform a large one given a vague prompt.
What if a model barely doesn’t fit?
Try a heavier quantization first — dropping one step (say, from Q5 to Q4) usually buys back more memory than it costs in quality. If it’s still tight, shorten how much context you keep around.
Is unified memory on a Mac the same as VRAM?
Close enough for planning purposes. It’s shared between the system and the model, so leave more headroom than you would on a dedicated GPU.