What to buy for local AI
Rule one: memory decides what fits, bandwidth decides speed. Pick the model you want first, then the hardware.
Best value used: RTX 3090 (24 GB)
24 GB runs 30B-class models at Q4. Measured ~158 tok/s (CUDA) on Llama 2 7B Q4_0 in the llama.cpp community scoreboard.
Check models on this hardware · Check price → (affiliate link slot, not active yet)
Fastest consumer: RTX 5090 (32 GB)
Top of the llama.cpp CUDA scoreboard; 32 GB still can't hold a 70B at Q4.
Check models on this hardware · Check price → (affiliate link slot, not active yet)
Budget 16 GB: RTX 5060 Ti 16GB
Enough for 14B models at Q4_K_M with room for context.
Check models on this hardware · Check price → (affiliate link slot, not active yet)
Big models, quiet: Mac with 64–128 GB
Unified memory fits 70B models; slower per token than top GPUs.
Check models on this hardware · Check price → (affiliate link slot, not active yet)
Disclosure: price links will become affiliate links. That never changes our fit verdicts.
Get the weekly "new models & what it takes to run them" email
Free. One email a week. Unsubscribe anytime.