Best local LLMs for 32 GB of VRAM
Short answer: the largest current models that fit fully in 32 GB are Kimi Linear 48B A3B (MoE) (IQ4_XS), Qwen3.6 35B A3B (MoE) (Q6_K), Qwen3.5 35B A3B (MoE) (Q6_K). 35 of 61 current models fit at Q3_K_M or better with an 8k context ESTIMATE.
Cards with 32 GB: RTX 5090, RTX PRO 4500 Blackwell, Tesla V100 32GB (PCIe / SXM2), Radeon AI Pro R9700.
| Model | Params | Best quant that fits | Needs (8k ctx) | Headroom |
|---|---|---|---|---|
| Kimi Linear 48B A3B (MoE) MoE | 49.1B | IQ4_XS | 28.3 GB | 3.7 GB |
| Qwen3.6 35B A3B (MoE) MoE | 36.0B | Q6_K | 31.7 GB | 0.3 GB |
| Qwen3.5 35B A3B (MoE) MoE | 36.0B | Q6_K | 31.7 GB | 0.3 GB |
| Olmo 3.1 32B | 32.2B | Q6_K | 29.7 GB | 2.3 GB |
| NVIDIA Nemotron 3.5 Lightning 30B A3B (MoE) MoE | 31.6B | Q6_K | 27.8 GB | 4.2 GB |
| NVIDIA Nemotron 3 Nano 30B A3B (MoE) MoE | 31.6B | Q6_K | 27.8 GB | 4.2 GB |
| Gemma 4 31B | 31.3B | Q6_K | 29.7 GB | 2.3 GB |
| GLM 4.7 Flash MoE | 31.2B | Q6_K | 27.9 GB | 4.1 GB |
| Qwen3 Coder 30B A3B (MoE) MoE | 30.5B | Q6_K | 27.7 GB | 4.3 GB |
| Qwen3 30B A3B (2507) (MoE) MoE | 30.5B | Q6_K | 27.7 GB | 4.3 GB |
| granite 4.2 30b | 29.3B | Q6_K | 28.0 GB | 4.0 GB |
| Qwen3.8 27B | 27.8B | Q6_K | 25.1 GB | 6.9 GB |
| Qwen3.6 27B | 27.8B | Q6_K | 25.1 GB | 6.9 GB |
| Qwen3.5 27B | 27.8B | Q6_K | 25.1 GB | 6.9 GB |
| Gemma 4 26B A4B (MoE) MoE | 25.8B | Q8_0 | 29.9 GB | 2.1 GB |
| Devstral Small 2 24B (2512) | 24.0B | Q8_0 | 28.7 GB | 3.3 GB |
| LFM2.5 24B A2B (MoE) MoE | 24.0B | Q8_0 | 27.5 GB | 4.5 GB |
| gpt-oss 20B (MoE) MoE | 20.9B | Q8_0 | 24.1 GB | 7.9 GB |
| Phi 4 reasoning vision 15B | 15.1B | Q8_0 | 19.1 GB | 12.9 GB |
| Ministral 3 14B (2512) | 13.9B | Q8_0 | 17.5 GB | 14.5 GB |
| Gemma 4 12B | 12.0B | Q8_0 | 14.8 GB | 17.2 GB |
| GLM 4.6V Flash | 10.3B | Q8_0 | 12.4 GB | 19.6 GB |
| Qwen3.5 9B | 9.7B | Q8_0 | 11.6 GB | 20.4 GB |
| Ministral 3 8B (2512) | 8.9B | Q8_0 | 11.7 GB | 20.3 GB |
| granite 4.2 8b | 8.8B | Q8_0 | 11.8 GB | 20.2 GB |
| Qwen3 VL 8B | 8.8B | Q8_0 | 11.6 GB | 20.4 GB |
| LFM2.5 8B A1B (MoE) MoE | 8.5B | Q8_0 | 10.1 GB | 21.9 GB |
| Gemma 4 E4B | 8.0B | Q8_0 | 9.7 GB | 22.3 GB |
| Gemma 4 E2B | 5.1B | Q8_0 | 6.4 GB | 25.6 GB |
| Qwen3.5 4B | 4.7B | Q8_0 | 6.1 GB | 25.9 GB |
| Qwen3 4B (2507) | 4.0B | Q8_0 | 6.3 GB | 25.7 GB |
| Nemotron Labs Dory 1.0 4B | 4.0B | Q8_0 | 5.5 GB | 26.5 GB |
| NVIDIA Nemotron 3 Nano 4B | 4.0B | Q8_0 | 5.2 GB | 26.8 GB |
| Ministral 3 3B (2512) | 3.8B | Q8_0 | 5.8 GB | 26.2 GB |
| Qwen3.5 2B | 2.3B | Q8_0 | 3.2 GB | 28.8 GB |
Best quant = highest of Q8_0, Q6_K, Q5_K_M, Q4_K_M, IQ4_XS, Q3_K_M that fits, f16 KV cache, 8k context. All memory figures are estimates (weights + KV cache + runtime buffers). Measured speeds per card are on the What fits my GPU? page and the checker.
FAQ
What is the biggest LLM that fits in 32 GB of VRAM?
By parameter count, Kimi Linear 48B A3B (MoE) at IQ4_XS (about 28.3 GB at 8k context, estimate). Bigger models need RAM offload or more GPUs.
How many current open models fit in 32 GB?
35 of 61 current models in our dataset fit fully on the GPU at Q3_K_M or better with 8k context (estimate).
Other tiers: 8 GB · 12 GB · 16 GB · 24 GB · 32 GB · 48 GB · 80 GB
Get the weekly "new models & what it takes to run them" email
Free. One email a week. Unsubscribe anytime.