A field guide to GGUF quant names
Q4_K_M, Q5_K_S, IQ2_XS — what the letters and numbers in a filename actually mean.
The short answer: the number tells you how compressed the model is — a smaller number means a smaller, faster file, with a little quality traded away. The letters tell you how that compression was done, sort of like the difference between a rushed suitcase-packing job and a smart one.
Think of it like packing a suitcase
Say you have to fit a whole wardrobe into one small suitcase. You could just cram everything in and let things get wrinkled — that’s a heavy, careless compression, like Q2 or Q3. Or you could pack carefully, keeping your favorite shirt nice and only letting the stuff you don’t care about get squashed — that’s a smarter compression, like the “K” quants below. Quantization is the same idea, just applied to a model’s “brain” instead of a suitcase.
The number: how compressed is it?
Q8 is barely compressed at all — closest to the original. Q4 is roughly half the size of Q8. The lower the number, the smaller and faster the model gets, but the more quality you risk losing. Below Q2, most people can actually notice the model getting worse at things like solving problems or writing carefully.
The letters: how good was the packing job?
“K” (as in Q4_K_M) means a smarter packing job. Instead of squeezing every part of the model equally, it protects the more important parts a little more and squeezes the less important parts harder. That’s why a “K” quant almost always feels better than an old-style quant of the same size.
The S/M/L at the end (small/medium/large) is just a slightly smaller or bigger version of that same packing method. M is the normal, “just right” choice most people pick.
“IQ” (as in IQ2_XS) is an even newer, even smarter packing method — mostly useful when you’re trying to squeeze a model down really small (2–3 bits). If you’re not desperate for space, you probably don’t need it.
Reading a full filename
Take a name like Llama-3.1-8B-Q4_K_M.gguf. Here’s what
each piece means:
Llama-3.1— the model’s name and version, like a car’s make and model.8B— how many “parameters” it has, or roughly how big its “brain” is. Here, 8 billion.Q4_K_M— the quantization: a smart, medium-sized compression.
Some filenames also carry the word “Instruct”, or end in “-it”. That means the model’s been specially trained to follow your instructions and chat back and forth, instead of just continuing a piece of text on its own. That’s a separate label from the quantization — which is exactly why every example on this page leaves it out, to keep one idea in view at a time.
A sensible default
If you’re not sure what to pick, go with Q4_K_M. It’s small enough to run comfortably, and rarely the reason a conversation feels slow. Step up to Q5_K_M or Q6_K if you’ve got memory to spare and want a bit more quality. Only go below Q3 if you truly have no other choice.
Common questions
What does it mean if a filename doesn’t say “Instruct”?
It’s a base model — trained to continue text rather than have a back-and-forth conversation with you. All the example filenames on this page are base models for exactly that reason.
Is a smaller quant ever the better choice?
Yes — if a bigger file means the model doesn’t fit in your computer’s memory at all. A smaller model running at full speed usually beats a bigger one crawling along ten times slower.
Do quantized models get worse the more I use them?
No. The compression happens once, when the file is made. Using it over and over doesn’t wear it down or make it worse.