Q4 vs Q8: which quantization should I use?
Q4_K_M for most people. Q8_0 needs about 1.75 times the weight memory (8.5 vs 4.85 bits per weight) for a small quality gain. In llama.cpp's published Llama 3 8B test, Q8_0 was nearly identical to F16 and Q4_K_M showed a small loss. Use Q8 only if it fits with your context.
Checked 2026-10-22 · memory figures are estimates
Memory needed (8k ctx, est.)
| Model | Q4_K_M | Q8_0 |
|---|---|---|
| Llama 3.1 8B Instruct | 6.8 GB | 10.6 GB |
| Qwen3 32B | 23.6 GB | 39.3 GB |
| Llama 3.3 70B Instruct | 48.2 GB | 80.6 GB |
Quality difference PUBLIC SOURCE
From the LLaMA 3 8B scoreboard in llama.cpp PR #6936 (Wikitext-2, lower is better): F16 perplexity 6.233, Q8_0 6.234, Q4_K_M 6.407. KL divergence from F16: Q8_0 0.0014, Q4_K_M 0.031. Quoted, not tested by us.
How we got the memory numbers
Weights = parameters × bits per weight ÷ 8, with Q4_K_M at 4.85 and Q8_0 at 8.5 bits, plus KV cache and buffers. Method.
FAQ
Is Q4 good enough for coding?
Q4_K_M is the common default. If a Q5 or Q6 fits with your context, it's a cheap step up. We haven't run quality tests ourselves.
What about Q5_K_M and Q6_K?
They sit between the two: 5.69 and 6.56 bits per weight. In the same llama.cpp test their quality loss was between Q8_0 and Q4_K_M.
Does the KV cache quant matter too?
Yes for memory. A q8_0 KV cache halves context memory. See How much VRAM does context use.
Open the checker · What fits my GPU?
More short answers
- What LLM can I run on my GPU?
- How much VRAM do I need for a 70B model?
- Is 16GB RAM enough to run an LLM locally?
- MLX vs GGUF on a Mac: which is faster?
- How much VRAM do I need for a 7B model?
- How much VRAM do I need for a 32B model?
- Is 12GB VRAM enough for local LLMs?
- How much VRAM does context length use?
- Can you run an LLM on CPU only?
Will It Run? Pro NOT BUILT YET
A personal setup for your exact hardware (Ollama / llama.cpp configs ready for your coding agent), an alert when a new model fits, and a one-click cloud fallback when it doesn't.
Not built yet. Join the waitlist, $10/mo planned, no charge now. What's planned
Email only. No payment info.