Best local LLMs for 24 GB of VRAM
Short answer: the largest current models that fit fully in 24 GB are Qwen3.6 35B A3B (MoE) (Q4_K_M), Qwen3.5 35B A3B (MoE) (Q4_K_M), Olmo 3.1 32B (Q4_K_M). 34 of 61 current models fit at Q3_K_M or better with an 8k context ESTIMATE.
Cards with 24 GB: RTX 4090, RTX 3090 Ti, RTX 3090, RTX 5090 Laptop, RTX PRO 4000 Blackwell, RTX PRO 4000 Blackwell SFF, RTX A5000, Tesla P40.
| Model | Params | Best quant that fits | Needs (8k ctx) | Headroom |
|---|---|---|---|---|
| Qwen3.6 35B A3B (MoE) MoE | 36.0B | Q4_K_M | 23.7 GB | 0.3 GB |
| Qwen3.5 35B A3B (MoE) MoE | 36.0B | Q4_K_M | 23.7 GB | 0.3 GB |
| Olmo 3.1 32B | 32.2B | Q4_K_M | 22.5 GB | 1.5 GB |
| NVIDIA Nemotron 3.5 Lightning 30B A3B (MoE) MoE | 31.6B | Q4_K_M | 20.8 GB | 3.2 GB |
| NVIDIA Nemotron 3 Nano 30B A3B (MoE) MoE | 31.6B | Q4_K_M | 20.8 GB | 3.2 GB |
| Gemma 4 31B | 31.3B | Q4_K_M | 22.7 GB | 1.3 GB |
| GLM 4.7 Flash MoE | 31.2B | Q4_K_M | 20.9 GB | 3.1 GB |
| Qwen3 Coder 30B A3B (MoE) MoE | 30.5B | Q4_K_M | 20.8 GB | 3.2 GB |
| Qwen3 30B A3B (2507) (MoE) MoE | 30.5B | Q4_K_M | 20.8 GB | 3.2 GB |
| granite 4.2 30b | 29.3B | Q4_K_M | 21.4 GB | 2.6 GB |
| Qwen3.8 27B | 27.8B | Q5_K_M | 21.9 GB | 2.1 GB |
| Qwen3.6 27B | 27.8B | Q5_K_M | 21.9 GB | 2.1 GB |
| Qwen3.5 27B | 27.8B | Q5_K_M | 21.9 GB | 2.1 GB |
| Gemma 4 26B A4B (MoE) MoE | 25.8B | Q6_K | 23.4 GB | 0.6 GB |
| Devstral Small 2 24B (2512) | 24.0B | Q6_K | 22.6 GB | 1.4 GB |
| LFM2.5 24B A2B (MoE) MoE | 24.0B | Q6_K | 21.4 GB | 2.6 GB |
| gpt-oss 20B (MoE) MoE | 20.9B | Q6_K | 18.8 GB | 5.2 GB |
| Phi 4 reasoning vision 15B | 15.1B | Q8_0 | 19.1 GB | 4.9 GB |
| Ministral 3 14B (2512) | 13.9B | Q8_0 | 17.5 GB | 6.5 GB |
| Gemma 4 12B | 12.0B | Q8_0 | 14.8 GB | 9.2 GB |
| GLM 4.6V Flash | 10.3B | Q8_0 | 12.4 GB | 11.6 GB |
| Qwen3.5 9B | 9.7B | Q8_0 | 11.6 GB | 12.4 GB |
| Ministral 3 8B (2512) | 8.9B | Q8_0 | 11.7 GB | 12.3 GB |
| granite 4.2 8b | 8.8B | Q8_0 | 11.8 GB | 12.2 GB |
| Qwen3 VL 8B | 8.8B | Q8_0 | 11.6 GB | 12.4 GB |
| LFM2.5 8B A1B (MoE) MoE | 8.5B | Q8_0 | 10.1 GB | 13.9 GB |
| Gemma 4 E4B | 8.0B | Q8_0 | 9.7 GB | 14.3 GB |
| Gemma 4 E2B | 5.1B | Q8_0 | 6.4 GB | 17.6 GB |
| Qwen3.5 4B | 4.7B | Q8_0 | 6.1 GB | 17.9 GB |
| Qwen3 4B (2507) | 4.0B | Q8_0 | 6.3 GB | 17.7 GB |
| Nemotron Labs Dory 1.0 4B | 4.0B | Q8_0 | 5.5 GB | 18.5 GB |
| NVIDIA Nemotron 3 Nano 4B | 4.0B | Q8_0 | 5.2 GB | 18.8 GB |
| Ministral 3 3B (2512) | 3.8B | Q8_0 | 5.8 GB | 18.2 GB |
| Qwen3.5 2B | 2.3B | Q8_0 | 3.2 GB | 20.8 GB |
Best quant = highest of Q8_0, Q6_K, Q5_K_M, Q4_K_M, IQ4_XS, Q3_K_M that fits, f16 KV cache, 8k context. All memory figures are estimates (weights + KV cache + runtime buffers). Measured speeds per card are on the What fits my GPU? page and the checker.
FAQ
What is the biggest LLM that fits in 24 GB of VRAM?
By parameter count, Qwen3.6 35B A3B (MoE) at Q4_K_M (about 23.7 GB at 8k context, estimate). Bigger models need RAM offload or more GPUs.
How many current open models fit in 24 GB?
34 of 61 current models in our dataset fit fully on the GPU at Q3_K_M or better with 8k context (estimate).
Other tiers: 8 GB · 12 GB · 16 GB · 24 GB · 32 GB · 48 GB · 80 GB
Get the weekly "new models & what it takes to run them" email
Free. One email a week. Unsubscribe anytime.