Cosmovex/tools

RTX 4060 8GB for local LLMs: which models fit and how fast they run

RTX 4060 8GB with Local AI Fit

This page opens with an RTX 4060 8GB and 16 GB of DDR5 RAM at 8K context. The usable memory on an 8 GB card is closer to 7.5 GB, which decides which models and quantisations fit entirely on the card. The list shows the fit and an estimated speed for each.

  1. Confirm the card and the amount of system RAM at the top.
  2. Look at which 7B and 8B models show a quantisation and a Fast badge.
  3. Tap Llama 3.1 8B to see why Q5_K_M is chosen instead of Q8_0.
  4. Try a 4K context to see what a shorter context frees up.
  5. Use "What do I need?" to see the memory an upgrade would require.
rtx 4060 8gb llm 8gb vram local llm4060 ollama modelsllama 8b 8gb vrambest quantisation for 8gb
Open Local AI Fit Free to start · no account

What to know

An 8 GB card has about 7.5 GB usable. Llama 3.1 8B at Q8_0 needs about 9.6 GB, so it does not fit; Q5_K_M needs about 7.0 GB at 8K context and does, with an estimated speed near 31 tokens per second. Mistral 7B fits at Q6_K (about 7.2 GB). Llama 3.2 3B fits at Q8_0 with plenty to spare, near 52 tokens per second, and the 1B model is very fast.

The 4060 has a narrow memory bus, about 272 GB/s, so it is slower than its VRAM size suggests when you compare it with a 3060 12GB at 360 GB/s. The extra four gigabytes on the older card let it hold bigger models, while the newer card generates faster on small ones.

A 14B model at Q4_K_M needs about 10.3 GB, so a quarter of it spills into system RAM. The estimate drops to about 13 tokens per second, still usable for chat. gpt-oss 20B at Q4_K_M needs about 12.9 GB; as a mixture-of-experts model it reads only about 3.6 billion parameters per token and stays near 31 tokens per second even with the spill. Dense 32B models and 70B models do not fit with 16 GB of RAM.

Updated · Cosmovex

Questions

What is the biggest model an 8 GB card can run?

Entirely on the card, an 8B model at Q5_K_M or a 7B model at Q6_K at 8K context. With part of the model in system RAM, 14B models at Q4_K_M and mixture-of-experts models such as gpt-oss 20B run at usable speeds if you have 16 GB of RAM or more.

Why does the tool pick Q5_K_M instead of Q8_0 for an 8B model?

Q8_0 needs about 9.6 GB with an 8K context, more than the 7.5 GB usable. Q5_K_M, at about 5.7 bits per weight, is the highest quality that fits entirely on the card; the quality difference from Q8_0 is small.

Does a shorter context help on 8 GB?

Yes, a little. The cache for Llama 3.1 8B is about 1 GB at 8K tokens and 0.5 GB at 4K. Going to 32K needs about 4 GB of cache, which pushes the same model into system RAM.

Is 16 GB of system RAM enough for offloading?

For 14B models yes, since only a few gigabytes spill over. For 32B models at Q4_K_M, which need about 21 GB in total, it is not enough once the system takes its 4 GB.