VRAM Calculator for Local LLMs
Estimate exact VRAM needed for model weights plus KV cache — the part most calculators skip, and the reason a model that loads fine at low context can still run out of memory once you ask it something long. Not sure which model to start with? See what your GPU can run →
Find these in the model's config.json on Hugging Face (num_hidden_layers, num_key_value_heads, head_dim).
K and V tensors have different compression sensitivity — V degrades quality faster than K at the same bit width. FP16/FP16 is the safe default.
For extended context via RoPE scaling, enter a custom value.
Leave at 1 for personal use (Ollama, LM Studio, llama.cpp). This is concurrent sequences, not llama.cpp's -b prompt-processing batch size — those are different things. Only increase if running a multi-user inference server where each user needs their own KV cache.
Find your card's memory bandwidth on TechPowerUp GPU Database.
Estimated VRAM usage
- Model weights
- —
- KV cache (total)
- —
- ↳ Keys
- —
- ↳ Values
- —
- Overhead (CUDA/framework)
- —
- Total
- —
ℹ Sliding window hybrid model — KV cache has two components:— sliding-window layers (fixed, capped at — tokens) +— global attention layers (grows with context length). The "max context" figure below reflects how much additional context the global layers can accommodate.
ℹ Hybrid attention model — KV cache calculated using— full-attention layers (out of — total). Linear/sliding-window layers don't accumulate a standard KV cache and are excluded from this estimate.
⚠ MoE model — weight VRAM is based on — total parameters. All experts must be resident in memory even though only —are active per token. KV cache is shown separately below and reflects this model's actual attention architecture.
Max context that fits in VRAM (weights + KV only):
—
Upper bound — real compute buffers grow with context, so usable context is lower, and decode speed drops sharply approaching the limit.
Compare max context at F16 / Q8 / Q4 KV cache →Estimated decode speed:
—
—
—
TurboQuant KV cache comparison
Theoretical estimates only — llama.cpp integration in progress, not yet in a stable release. Values assume both K and V compressed at the same bit target with no codebook overhead.
| Method | KV cache size | vs F16/F16 |
|---|---|---|
| Enter model parameters above to see estimates. | ||
TurboQuant paper (ICLR 2026)— K and V have different sensitivity to compression; real deployments may use asymmetric bit targets rather than the symmetric estimates shown here.
How this is calculated
Total VRAM is the sum of three components, calculated separately rather than estimated as a single multiplier:
Model weights
Parameter count × effective bytes-per-parameter for the chosen quantization. GGUF K-quants use mixed precision and block-wise scale factors, so this isn't a flat bit-count division — the values used here are calibrated against real GGUF file sizes.
KV cache
2 × layers × KV heads × head dimension × context length × 2 bytes × batch size. This is the term that grows with context length and is usually why a model that loads fine at short context runs out of memory on long conversations.
Overhead
A fixed ~0.6GB buffer for CUDA context initialization and inference framework overhead. Actual overhead varies slightly by backend (llama.cpp, vLLM, etc.).
Verified against a real Qwen3-14B Q8_0 load on an RTX 3060 12GB: predicted weight size alone (≈15.7GB) correctly exceeds 12GB, matching the observed out-of-memory result, while Q4_K_M (≈9.0GB) correctly fits, leaving modest headroom for context.
Getting a different number from another VRAM calculator?
That's common — different tools use different methodology for quantization sizing and KV cache. Here's a breakdown of why the numbers diverge.
FAQ
Why does this calculator give different numbers than other VRAM calculators?
Most calculators only account for model weight size and ignore KV cache, which scales with context length. A model that fits comfortably at 2K context can run out of memory at 32K context purely from KV cache growth. This calculator separates both terms so you can see exactly where your VRAM is going.
What's the difference between Q4_K_M and Q8_0?
Q4_K_M uses roughly 0.55 bytes per parameter with mixed-precision blocks, producing smaller files with a modest quality tradeoff. Q8_0 uses roughly 1 byte per parameter and is close to lossless compared to full FP16 precision. Q4_K_M is the practical default for most consumer GPUs; Q8_0 is worth it if you have VRAM to spare.
Does KV cache quantization change these numbers?
This calculator assumes KV cache stored at FP16 (2 bytes per value), which is the common default in llama.cpp and Ollama. Some setups support quantized KV cache (e.g. Q8 or Q4 KV cache) to reduce this further, which would lower the KV cache portion of the estimate proportionally.