Cosmovex/tools

Which local LLMs run on an RTX 3060 12GB? Model sizes, quantisation and speed

RTX 3060 12GB with Local AI Fit

This page opens with an RTX 3060 12GB and 32 GB of DDR5 RAM at 8K context. It lists every model with the quantisation that fits, whether it sits entirely on the card, and an estimated speed. Change the RAM or context to match your machine.

  1. Check the card and RAM in the box at the top; change them if your machine differs.
  2. Read the Fast, Usable, Slow and Won't fit badges down the list.
  3. Tap a row to see weights, context cache and buffers added up.
  4. Raise the context length to see which models stop fitting when you give them longer documents.
  5. Open "What do I need?" to see what hardware a bigger model would take.
rtx 3060 12gb llm 3060 12gb local llmrtx 3060 ollamavram for 14b modelllama 8b on 12gb vram
Open Local AI Fit Free to start · no account

What to know

A 12 GB card has about 11.3 GB usable once the display and driver take their share. That is enough to hold Llama 3.1 8B at Q8_0 (about 9.6 GB with an 8K cache, estimated near 27 tokens per second) and Qwen3 14B or Phi-4 14B at Q4_K_M (about 10 to 11 GB, near 26 tokens per second). Those are the comfortable fits: the whole model sits in GPU memory.

Models slightly larger than the card still work. gpt-oss 20B at Q4_K_M needs about 12.9 GB, so roughly 88% sits on the card and the rest in system RAM. Because it is a mixture-of-experts model that reads only about 3.6 billion parameters per token, the estimate stays high, around 55 tokens per second. Dense models behave differently: Qwen3 32B at Q4_K_M needs about 21 GB, so almost half of it is read from RAM on every token and the estimate falls to about 5 tokens per second.

The wall is 70B-class models. Llama 3.3 70B at Q4_K_M needs about 43 GB. Even with 32 GB of RAM, that is more than the 11.3 GB on the card plus the roughly 28 GB of RAM left after the system takes 4 GB, so it does not fit at all. Cutting to Q2 would not make it a good experience, and the quality loss is large; a 14B or 30B-class model is the realistic ceiling for this card.

Updated · Cosmovex

Questions

Is 12 GB of VRAM enough for local LLMs?

It is enough for models up to about 14B parameters at Q4_K_M, or an 8B model at Q8_0, with room for a normal context. It is not enough for 32B or 70B models without sending much of the model to system RAM, which is several times slower.

Can I run Llama 3 70B on an RTX 3060 12GB?

Not in a practical way. At Q4_K_M it needs about 43 GB including the context cache. The card holds about 11 GB and the system RAM can hold the rest only if you have 48 GB or more, and then generation drops to roughly one token per second or less.

Does the 3060's memory speed matter?

Yes. Generation speed is limited by how fast the weights can be read, and the 3060 has about 360 GB/s of memory bandwidth. That is why a 4090 with 1,008 GB/s is roughly three times faster on the same model even when both hold it entirely.

Which quantisation should I choose on 12 GB?

Use Q8_0 for 7B and 8B models, which fit with room to spare, and Q4_K_M for 14B models. If a 14B model at Q4_K_M does not fit with your context length, reduce the context before you drop to Q3, which lowers quality.