Short answers

How much VRAM does context length use?

KV cache for Llama 3.1 8B Instruct (est.)

How we got this number

KV cache = 2 (keys and values) × layers × KV heads × head dimension × context × bytes per element. Llama 3.1 8B Instruct has 32 layers, 8 KV heads and head dimension 128 (config.json), so f16 at 32k is 2 × 32 × 8 × 128 × 32768 × 2 bytes ≈ 4.3 GB. Models with sliding-window or MLA attention need less; the checker handles those.

Llama 3.1 8B Instruct page · Try a context length in the checker

FAQ

Q4 vs Q8 KV cache: which should I use?

q8_0 halves KV memory and is the usual first step. q4_0 saves more but may cost more quality; we haven't measured that. Set it in llama.cpp with -ctk and -ctv.

How much VRAM does 256k context use?

It depends on the architecture. Models with few KV heads or hybrid attention need far less per token. Pick the model and 256k in the checker to see the estimate.

Does context change which quant fits?

Yes. A model that fits at 8k can fail at 128k, so the checker and What fits pages always state the context they assume.

Open the checker · What fits my GPU?

More short answers

• What LLM can I run on my GPU?
• How much VRAM do I need for a 70B model?
• Is 16GB RAM enough to run an LLM locally?
• MLX vs GGUF on a Mac: which is faster?
• How much VRAM do I need for a 7B model?
• How much VRAM do I need for a 32B model?
• Is 12GB VRAM enough for local LLMs?
• Can you run an LLM on CPU only?
• Q4 vs Q8: which quantization should I use?