Can I run Mistral Small 3.1 24B locally?

Mistral Small 3.1 24B has 24.01 billion parameters (Hugging Face). At the popular Q4_K_M quantization it needs roughly 16.6 GB of GPU memory ESTIMATE. It has been on Hugging Face since 2025-03-11. This is an older model; see the current list for newer options.

Memory needed by quantization

Quant	Weights	Total needed (est.)
IQ1_S	4.7 GB	6.2 GB
IQ2_XXS	6.2 GB	7.8 GB
IQ2_M	8.1 GB	9.8 GB
Q2_K	9.5 GB	11.3 GB
IQ3_XXS	9.2 GB	10.9 GB
Q3_K_S	11.0 GB	12.8 GB
IQ3_M	11.0 GB	12.8 GB
Q3_K_M	12.0 GB	13.9 GB
IQ4_XS	12.8 GB	14.7 GB
Q4_0	13.5 GB	15.5 GB
Q4_K_M	14.6 GB	16.6 GB
MXFP4	13.9 GB	15.9 GB
Q5_K_M	17.1 GB	19.2 GB
Q6_K	19.7 GB	21.9 GB
Q8_0	25.5 GB	28.1 GB
F16	48.0 GB	51.7 GB

KV cache by context length ESTIMATE

Context	f16	q8_0	q4_0
4k	0.67 GB	0.36 GB	0.19 GB
8k	1.34 GB	0.71 GB	0.38 GB
16k	2.68 GB	1.43 GB	0.75 GB
32k	5.37 GB	2.85 GB	1.51 GB
64k	10.74 GB	5.70 GB	3.02 GB
128k	21.47 GB	11.41 GB	6.04 GB

From config.json: 40 KV-caching attention layers, 8 KV heads × 128 head dim. Max context 128k. The tables above assume 4k context.

Which GPUs and Macs can run it?

Hardware	Memory	Q4_K_M	Q8_0	F16
RTX 5090	32 GB	✅	✅	❌
RTX 5080	16 GB	❌	❌	❌
RTX 5070 Ti	16 GB	❌	❌	❌
RTX 5070	12 GB	❌	❌	❌
RTX 5060 Ti 16GB	16 GB	❌	❌	❌
RTX 5060 Ti 8GB	8 GB	❌	❌	❌
RTX 5060	8 GB	❌	❌	❌
RTX 5050	8 GB	❌	❌	❌
RTX 4090	24 GB	✅	❌	❌
RTX 4080 Super	16 GB	❌	❌	❌
RTX 4080	16 GB	❌	❌	❌
RTX 4070 Ti Super	16 GB	❌	❌	❌
RTX 4070 Ti	12 GB	❌	❌	❌
RTX 4070 Super	12 GB	❌	❌	❌
RTX 4070	12 GB	❌	❌	❌
RTX 4060 Ti 16GB	16 GB	❌	❌	❌
RTX 4060 Ti 8GB	8 GB	❌	❌	❌
RTX 4060	8 GB	❌	❌	❌
RTX 3090 Ti	24 GB	✅	❌	❌
RTX 3090	24 GB	✅	❌	❌
RTX 3080 Ti	12 GB	❌	❌	❌
RTX 3080 12GB	12 GB	❌	❌	❌
RTX 3080 10GB	10 GB	❌	❌	❌
RTX 3070 Ti	8 GB	❌	❌	❌
RTX 3070	8 GB	❌	❌	❌
RTX 3060 Ti	8 GB	❌	❌	❌
RTX 3060 12GB	12 GB	❌	❌	❌
RTX 3060 8GB	8 GB	❌	❌	❌
RTX 3050 8GB	8 GB	❌	❌	❌
RTX 3050 6GB	6 GB	❌	❌	❌
RTX 2080 Ti	11 GB	❌	❌	❌
RTX 2080 Super	8 GB	❌	❌	❌
RTX 2080	8 GB	❌	❌	❌
RTX 2070 Super	8 GB	❌	❌	❌
RTX 2070	8 GB	❌	❌	❌
RTX 2060 Super	8 GB	❌	❌	❌
RTX 2060 6GB	6 GB	❌	❌	❌
RTX 2060 12GB	12 GB	❌	❌	❌
GTX 1080 Ti	11 GB	❌	❌	❌
GTX 1660 Super	6 GB	❌	❌	❌
RTX 5090 Laptop	24 GB	✅	❌	❌
RTX 4090 Laptop	16 GB	❌	❌	❌
RTX PRO 6000 Blackwell	96 GB	✅	✅	✅
RTX 6000 Ada	48 GB	✅	✅	❌
RTX A6000	48 GB	✅	✅	❌
RTX A5000	24 GB	✅	❌	❌
RTX A4000	16 GB	❌	❌	❌
Tesla P40	24 GB	✅	❌	❌
DGX Spark (GB10)	128 GB	✅	✅	✅
Radeon RX 9070 XT	16 GB	❌	❌	❌
Radeon RX 9070	16 GB	❌	❌	❌
Radeon RX 9060 XT 16GB	16 GB	❌	❌	❌
Radeon RX 9060 XT 8GB	8 GB	❌	❌	❌
Radeon RX 7900 XTX	24 GB	✅	❌	❌
Radeon RX 7900 XT	20 GB	✅	❌	❌
Radeon RX 7900 GRE	16 GB	❌	❌	❌
Radeon RX 7800 XT	16 GB	❌	❌	❌
Radeon RX 7700 XT	12 GB	❌	❌	❌
Radeon RX 7600 XT	16 GB	❌	❌	❌
Radeon RX 7600	8 GB	❌	❌	❌
Radeon RX 6950 XT	16 GB	❌	❌	❌
Radeon RX 6900 XT	16 GB	❌	❌	❌
Radeon RX 6800 XT	16 GB	❌	❌	❌
Radeon RX 6800	16 GB	❌	❌	❌
Radeon RX 6750 XT	12 GB	❌	❌	❌
Radeon RX 6700 XT	12 GB	❌	❌	❌
Radeon RX 6650 XT	8 GB	❌	❌	❌
Radeon RX 6600 XT	8 GB	❌	❌	❌
Radeon RX 6600	8 GB	❌	❌	❌
Radeon AI Pro R9700	32 GB	✅	✅	❌
Intel Arc B580	12 GB	❌	❌	❌
Intel Arc B570	10 GB	❌	❌	❌
Intel Arc A770 16GB	16 GB	❌	❌	❌
Intel Arc A770 8GB	8 GB	❌	❌	❌
Intel Arc A750	8 GB	❌	❌	❌
Intel Arc Pro B60	24 GB	✅	❌	❌
GTX 1650	4 GB	❌	❌	❌
GTX 1650 Super	4 GB	❌	❌	❌
GTX 1650 Laptop	4 GB	❌	❌	❌
GTX 1050 Ti	4 GB	❌	❌	❌
RTX 3050 Laptop 4GB	4 GB	❌	❌	❌
RTX 2050 Laptop	4 GB	❌	❌	❌
Quadro T1000	4 GB	❌	❌	❌
RX 6500 XT 4GB	4 GB	❌	❌	❌
RX 6400	4 GB	❌	❌	❌
Intel Arc A380	6 GB	❌	❌	❌
Apple M1 (16 GB)	16 GB	❌	❌	❌
Apple M1 Pro (16 GB)	16 GB	❌	❌	❌
Apple M1 Pro (32 GB)	32 GB	✅	❌	❌
Apple M1 Max (32 GB)	32 GB	✅	❌	❌
Apple M1 Max (64 GB)	64 GB	✅	✅	❌
Apple M1 Ultra (64 GB)	64 GB	✅	✅	❌
Apple M1 Ultra (128 GB)	128 GB	✅	✅	✅
Apple M2 (16 GB)	16 GB	❌	❌	❌
Apple M2 (24 GB)	24 GB	❌	❌	❌
Apple M2 Pro (16 GB)	16 GB	❌	❌	❌
Apple M2 Pro (32 GB)	32 GB	✅	❌	❌
Apple M2 Max (32 GB)	32 GB	✅	❌	❌
Apple M2 Max (64 GB)	64 GB	✅	✅	❌
Apple M2 Max (96 GB)	96 GB	✅	✅	✅
Apple M2 Ultra (64 GB)	64 GB	✅	✅	❌
Apple M2 Ultra (128 GB)	128 GB	✅	✅	✅
Apple M2 Ultra (192 GB)	192 GB	✅	✅	✅
Apple M3 (16 GB)	16 GB	❌	❌	❌
Apple M3 (24 GB)	24 GB	❌	❌	❌
Apple M3 Pro (36 GB)	36 GB	✅	❌	❌
Apple M3 Max (36 GB)	36 GB	✅	❌	❌
Apple M3 Max (48 GB)	48 GB	✅	✅	❌
Apple M3 Max (64 GB)	64 GB	✅	✅	❌
Apple M3 Max (96 GB)	96 GB	✅	✅	✅
Apple M3 Max (128 GB)	128 GB	✅	✅	✅
Apple M3 Ultra (96 GB)	96 GB	✅	✅	✅
Apple M3 Ultra (256 GB)	256 GB	✅	✅	✅
Apple M3 Ultra (512 GB)	512 GB	✅	✅	✅
Apple M4 (16 GB)	16 GB	❌	❌	❌
Apple M4 (24 GB)	24 GB	❌	❌	❌
Apple M4 (32 GB)	32 GB	✅	❌	❌
Apple M4 Pro (24 GB)	24 GB	❌	❌	❌
Apple M4 Pro (48 GB)	48 GB	✅	✅	❌
Apple M4 Pro (64 GB)	64 GB	✅	✅	❌
Apple M4 Max (36 GB)	36 GB	✅	❌	❌
Apple M4 Max (48 GB)	48 GB	✅	✅	❌
Apple M4 Max (64 GB)	64 GB	✅	✅	❌
Apple M4 Max (128 GB)	128 GB	✅	✅	✅
Apple M5 (16 GB)	16 GB	❌	❌	❌
Apple M5 (24 GB)	24 GB	❌	❌	❌
Apple M5 (32 GB)	32 GB	✅	❌	❌
Apple M5 Pro (24 GB)	24 GB	❌	❌	❌
Apple M5 Pro (48 GB)	48 GB	✅	✅	❌
Apple M5 Pro (64 GB)	64 GB	✅	✅	❌
Apple M5 Max (36 GB)	36 GB	✅	❌	❌
Apple M5 Max (48 GB)	48 GB	✅	✅	❌
Apple M5 Max (64 GB)	64 GB	✅	✅	❌
Apple M5 Max (128 GB)	128 GB	✅	✅	✅
AMD Ryzen AI Max+ 395 Strix Halo 32GB (Linux, ~24 GB for GPU) · Radeon 8060S · Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini G1a	24 GB	✅	❌	❌
AMD Ryzen AI Max+ 395 Strix Halo 32GB (Windows, ~24 GB for GPU) · Radeon 8060S · Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini G1a	24 GB	✅	❌	❌
AMD Ryzen AI Max+ 395 Strix Halo 64GB (Linux, ~56 GB for GPU) · Radeon 8060S · Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini G1a	56 GB	✅	✅	⚠️ tight
AMD Ryzen AI Max+ 395 Strix Halo 64GB (Windows, ~48 GB for GPU) · Radeon 8060S · Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini G1a	48 GB	✅	✅	❌
AMD Ryzen AI Max+ 395 Strix Halo 96GB (Linux, ~88 GB for GPU) · Radeon 8060S · Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini G1a	88 GB	✅	✅	✅
AMD Ryzen AI Max+ 395 Strix Halo 96GB (Windows, ~72 GB for GPU) · Radeon 8060S · Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini G1a	72 GB	✅	✅	✅
AMD Ryzen AI Max+ 395 Strix Halo 128GB (Linux, ~120 GB for GPU) · Radeon 8060S · Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini G1a	120 GB	✅	✅	✅
AMD Ryzen AI Max+ 395 Strix Halo 128GB (Windows, ~96 GB for GPU) · Radeon 8060S · Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini G1a	96 GB	✅	✅	✅
AMD Ryzen AI Max 390/385 Strix Halo 32GB (Linux, ~24 GB for GPU) · Radeon 8050S · Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini G1a	24 GB	✅	❌	❌
AMD Ryzen AI Max 390/385 Strix Halo 32GB (Windows, ~24 GB for GPU) · Radeon 8050S · Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini G1a	24 GB	✅	❌	❌
AMD Ryzen AI Max 390/385 Strix Halo 64GB (Linux, ~56 GB for GPU) · Radeon 8050S · Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini G1a	56 GB	✅	✅	⚠️ tight
AMD Ryzen AI Max 390/385 Strix Halo 64GB (Windows, ~48 GB for GPU) · Radeon 8050S · Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini G1a	48 GB	✅	✅	❌
AMD Ryzen AI Max 390/385 Strix Halo 128GB (Linux, ~120 GB for GPU) · Radeon 8050S · Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini G1a	120 GB	✅	✅	✅
AMD Ryzen AI Max 390/385 Strix Halo 128GB (Windows, ~96 GB for GPU) · Radeon 8050S · Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini G1a	96 GB	✅	✅	✅

✅ fits · ⚠️ fits with little headroom · ❌ doesn't fit on this one device (multiple GPUs or RAM offload may work; check your setup). Apple Silicon assumes macOS lets the GPU use ~67–75% of unified memory. All fit verdicts are estimates.

Get the weekly "new models & what it takes to run them" email

Free. One email a week. Unsubscribe anytime.