Can I run DeepSeek R1 Distill Qwen 7B locally?
DeepSeek R1 Distill Qwen 7B has 7.62 billion parameters (Hugging Face). At the popular Q4_K_M quantization it needs roughly 5.7 GB of GPU memory ESTIMATE. It has been on Hugging Face since 2025-01-20. This is an older model; see the current list for newer options.
Memory needed by quantization
| Quant | Weights | Total needed (est.) |
|---|---|---|
| IQ1_S | 1.5 GB | 2.4 GB |
| IQ2_XXS | 2.0 GB | 2.9 GB |
| IQ2_M | 2.6 GB | 3.5 GB |
| Q2_K | 3.0 GB | 4.0 GB |
| IQ3_XXS | 2.9 GB | 3.9 GB |
| Q3_K_S | 3.5 GB | 4.5 GB |
| IQ3_M | 3.5 GB | 4.5 GB |
| Q3_K_M | 3.8 GB | 4.8 GB |
| IQ4_XS | 4.0 GB | 5.1 GB |
| Q4_0 | 4.3 GB | 5.3 GB |
| Q4_K_M | 4.6 GB | 5.7 GB |
| MXFP4 | 4.4 GB | 5.5 GB |
| Q5_K_M | 5.4 GB | 6.5 GB |
| Q6_K | 6.2 GB | 7.4 GB |
| Q8_0 | 8.1 GB | 9.3 GB |
| F16 | 15.2 GB | 16.8 GB |
KV cache by context length ESTIMATE
| Context | f16 | q8_0 | q4_0 |
|---|---|---|---|
| 4k | 0.23 GB | 0.12 GB | 0.07 GB |
| 8k | 0.47 GB | 0.25 GB | 0.13 GB |
| 16k | 0.94 GB | 0.50 GB | 0.26 GB |
| 32k | 1.88 GB | 1.00 GB | 0.53 GB |
| 64k | 3.76 GB | 2.00 GB | 1.06 GB |
| 128k | 7.52 GB | 3.99 GB | 2.11 GB |
From config.json: 28 KV-caching attention layers, 4 KV heads × 128 head dim. Max context 128k. The tables above assume 4k context.
Which GPUs and Macs can run it?
✅ fits · ⚠️ fits with little headroom · ❌ doesn't fit on this one device (multiple GPUs or RAM offload may work; check your setup). Apple Silicon assumes macOS lets the GPU use ~67–75% of unified memory. All fit verdicts are estimates.
Get the weekly "new models & what it takes to run them" email
Free. One email a week. Unsubscribe anytime.