RTX 3060 12GB for Local LLMs: What Fits, What Doesn't, and How Fast
The RTX 3060 12GB is one of the most common local LLM cards — 12GB of VRAM at a mid-range price, with enough headroom to run serious models. Here's exactly what fits, at what quantization, and at what context length, with real numbers rather than rough estimates.
The hardware baseline
The RTX 3060 12GB has 360 GB/s of memory bandwidth — the number that actually determines decode speed for local LLMs. Model weight size determines what fits; bandwidth determines how fast tokens generate once it's loaded. Both matter, and they're independent of each other.
At roughly 70% real-world utilization of peak bandwidth (accounting for KV cache reads, framework overhead, and memory controller efficiency), effective bandwidth is around 252 GB/s for decode. That puts the 3060 solidly in "fast enough for interactive use" territory for models up to about 14B parameters at Q4 quantization.
What fits — model by model
Weight sizes below are calculated from real GGUF file sizes rather than theoretical bit counts. K-quants carry embedding tables, output heads, and per-block scale factors that naive bit-math misses — the effective bytes per parameter at Q4_K_M is closer to 0.61 than the 0.5 bytes (4 bits ÷ 8) that a flat calculation gives.
| Model | Quant | Weight size | Fits 12GB | Est. speed |
|---|---|---|---|---|
| Llama 3.1 8B | Q4_K_M | 4.90 GB | ✓ | ~51 tok/s |
| Llama 3.1 8B | Q8_0 | 8.51 GB | ✓ | ~30 tok/s |
| Qwen3 8B | Q4_K_M | 5.00 GB | ✓ | ~50 tok/s |
| Qwen3 8B | Q8_0 | 8.69 GB | ✓ | ~29 tok/s |
| Gemma 2 9B | Q4_K_M | 5.64 GB | ✓ | ~45 tok/s |
| Gemma 2 9B | Q8_0 | 9.79 GB | ✓ | ~26 tok/s |
| Phi-4 14B | Q4_K_M | 8.97 GB | ✓ | ~28 tok/s |
| Phi-4 14B | Q5_K_M | 10.44 GB | ✓ | ~24 tok/s |
| Qwen3 14B | Q4_K_M | 9.03 GB | ✓ | ~28 tok/s |
| Qwen3 14B | Q5_K_M | 10.51 GB | ✓ | ~24 tok/s |
| Mistral Small 24B | Q4_K_M | 14.64 GB | ✗ | — |
| Qwen3 32B | Q4_K_M | 20.01 GB | ✗ | — |
Weight sizes calculated from calibrated bytes-per-parameter values sourced from real GGUF file sizes. Speed estimates assume 360 GB/s peak bandwidth at 70% real-world efficiency. Actual speed varies by backend and context length. Treat speeds as ±20%.
Context length: the hidden constraint
Whether a model fits isn't just about weight size — KV cache grows with context length and comes out of the same 12GB pool. This catches people off guard: a model loads fine, then OOMs mid-conversation once the context gets long enough.
For Qwen3 14B at Q4_K_M (9.03GB weights), here's how context length changes the total VRAM requirement:
| Context length | KV cache | Total (weights + KV + overhead) | Fits 12GB |
|---|---|---|---|
| 4K (4,096) | 0.67 GB | 10.34 GB | ✓ |
| 8K (8,192) | 1.34 GB | 11.01 GB | ✓ |
| 16K (16,384) | 2.68 GB | 12.36 GB | ✗ |
| 32K (32,768) | 5.37 GB | 15.04 GB | ✗ |
The practical limit for Qwen3-14B Q4_K_M on a 3060 12GB is 8K context. If you need longer context with a 14B model, dropping to Q4_K_S (slightly smaller file, minor quality tradeoff) buys a bit more headroom. Alternatively, an 8B model at the same quant leaves enough room to push to 32K context comfortably.
Recommended configurations
Best overall: Qwen3 14B Q4_K_M
9.03GB weights, fits at up to 8K context, ~28 tok/s. The current sweet spot for the 3060 — a genuinely capable model at a speed that feels responsive. Qwen3's instruction following and reasoning hold up well at Q4_K_M.
Long context: Llama 3.1 8B Q8_0
8.51GB weights, leaves enough room for 32K+ context, ~30 tok/s. If you need long conversations or document processing, the 8B at Q8_0 (near-lossless) is a better choice than 14B at Q4 — the quality difference between model sizes is smaller than the quality difference between Q4 and Q8 at this scale.
High quality, max VRAM: Phi-4 14B Q5_K_M
10.44GB weights, fits at 4–8K context, ~24 tok/s. Phi-4 punches above its parameter count for reasoning and coding tasks. Q5_K_M gives noticeably better quality than Q4 with a modest size increase. Context headroom is limited — watch your conversation length.
Check any model or quant combination
The table above covers the most common configurations, but you can check any model, quantization, and context length combination — including KV cache quantization and estimated decode speed for your specific GPU — using the calculator.
FAQ
Can the RTX 3060 12GB run Qwen3-14B?
Yes — at Q4_K_M quantization, Qwen3-14B requires about 9.03GB for weights plus roughly 1.34GB of KV cache at 8K context, totaling around 11.0GB. That fits within 12GB with about 1.0GB headroom. At Q5_K_M it takes about 10.5GB for weights, which still fits at short context but leaves less room. Q8_0 at 15.7GB does not fit.
What is the largest model that fits on a 3060 12GB?
At Q4_K_M quantization, models up to about 14–15B parameters fit comfortably. Phi-4 14B and Qwen3 14B both fit at Q4_K_M (around 9.0GB for weights). Mistral Small 24B at Q4_K_M requires about 14.6GB — past what fits on 12GB. You can push further with more aggressive quantization (IQ3_XXS for a 14B model takes about 6.0GB) but quality degrades noticeably below Q4.
How fast does the RTX 3060 run local LLMs?
The RTX 3060 12GB has 360 GB/s memory bandwidth, which is the primary factor for decode speed. At Q4_K_M quantization, 8B models run at roughly 50 tok/s and 14B models run at roughly 28 tok/s — both fast enough for comfortable interactive use. These are estimates based on bandwidth utilization; actual speed varies by backend (llama.cpp, Ollama) and context length.
Why does context length affect whether a model fits?
Model weight size is fixed once you pick a quantization level. KV cache memory grows with context length — for Qwen3-14B at Q4_K_M, each additional 8K tokens of context adds about 1.34GB of KV cache. At 8K context the total is about 11.0GB (fits). At 16K it's about 12.4GB (doesn't fit on a 12GB card). This is why choosing a context length matters as much as choosing a quant level.