How much VRAM do I need for a 7B model?
About 6.3 GB at Q4_K_M with 8k context (est.), based on Mistral 7B Instruct v0.3. So an 8 GB card like the RTX 4060 runs it, and 12 GB leaves room for Q8 or longer context. This is for running the model. Training or fine-tuning needs far more and is out of scope here.
Checked 2026-10-22 · memory figures are estimates
| Quant | Weights | Total needed (est., 8k ctx) |
|---|---|---|
| Q3_K_M | 3.6 GB | 5.5 GB |
| IQ4_XS | 3.9 GB | 5.7 GB |
| Q4_K_M | 4.4 GB | 6.3 GB |
| Q5_K_M | 5.2 GB | 7.1 GB |
| Q6_K | 5.9 GB | 7.9 GB |
| Q8_0 | 7.7 GB | 9.8 GB |
| F16 | 14.5 GB | 16.9 GB |
How we got this number
Memory needed = parameters × bits per weight ÷ 8 (Q4_K_M is 4.85 bits, Q8_0 is 8.5), plus the KV cache for your context and runtime buffers. Worked example: Mistral 7B Instruct v0.3 has 7.25B parameters, so Q4_K_M weights are about 4.4 GB; with an 8k f16 KV cache (1.1 GB) and buffers the total is about 6.3 GB. All estimates. Full method.
Mistral 7B Instruct v0.3 page · 8 GB VRAM list · Check an RTX 4060
FAQ
Can a 6 GB GPU run a 7B model?
Only at lower quants or short context. Q4_K_M needs about 5.8 GB at 4k (est.), so 6 GB is tight. Q3_K_M needs about 4.9 GB.
How much for Q8 7B?
About 9.8 GB at Q8_0 with 8k context (est.), so a 12 GB card.
What about fine-tuning a 7B model?
Training needs far more memory than running, and we don't estimate it. Our numbers cover inference only.
Open the checker · What fits my GPU?
More short answers
- What LLM can I run on my GPU?
- How much VRAM do I need for a 70B model?
- Is 16GB RAM enough to run an LLM locally?
- MLX vs GGUF on a Mac: which is faster?
- How much VRAM do I need for a 32B model?
- Is 12GB VRAM enough for local LLMs?
- How much VRAM does context length use?
- Can you run an LLM on CPU only?
- Q4 vs Q8: which quantization should I use?
Will It Run? Pro NOT BUILT YET
A personal setup for your exact hardware (Ollama / llama.cpp configs ready for your coding agent), an alert when a new model fits, and a one-click cloud fallback when it doesn't.
Not built yet. Join the waitlist, $10/mo planned, no charge now. What's planned
Email only. No payment info.