RTX 4090 24GB with Local AI Fit
This page opens with an RTX 4090 24GB and 64 GB of DDR5 RAM at 8K context. At about 22.6 GB usable and 1,008 GB/s of bandwidth, it holds 32B-class models entirely and shows exactly where 70B models stop being practical.
- Check the card and RAM at the top.
- Read down the list: the Fast rows are entirely on the card.
- Tap Qwen3 32B and note the memory it needs against the 22.6 GB usable.
- Raise the context to 32K and watch which rows change quantisation.
- Open Llama 3.3 70B to see how much of it ends up in system RAM.
What to know
With about 22.6 GB usable, the 4090 holds Qwen3 14B and Phi-4 14B at Q8_0 (about 17 GB, near 42 tokens per second), gpt-oss 20B at Q8_0 (about 22 GB, high as a mixture-of-experts model), and Qwen3 32B at Q4_K_M (about 21 GB, near 33 tokens per second). The 32B at Q4 is the largest dense model that is comfortable here.
Llama 3.3 70B at Q4_K_M needs about 43 GB at 8K context. About half sits on the card and half in RAM, and because every token reads the RAM half over a path of roughly 90 GB/s, the estimate collapses to about 2.6 tokens per second. That is readable for a patient batch job and not for conversation. Mixtral 8x7B, with about 12.9 billion parameters read per token, does far better at about 20 tokens per second despite needing 28 GB.
Longer context eats headroom quickly. Qwen3 32B at Q4_K_M with a 32K context needs about 27 GB, of which 8 GB is cache, so it no longer fits on the card alone and the tool drops to a lower-quality option or a split. If you want long documents with a 32B model, plan for a 32 GB card or a Mac with unified memory.
Updated · Cosmovex
Questions
Can an RTX 4090 run a 70B model?
At Q4_K_M it needs about 43 GB including the cache, so it cannot be held on a 24 GB card. With 64 GB of system RAM it runs split between the card and RAM at roughly 2 to 3 tokens per second by this estimate. At Q2 it would fit more of the model on the card but the quality loss is large.
What is the largest model that fits entirely on 24 GB?
A 32B dense model at Q4_K_M, about 21 GB with an 8K cache, or a mixture-of-experts model such as gpt-oss 20B at Q8_0 or Qwen3 30B-A3B at Q5_K_M. Anything above that has to share system RAM.
Is a second RTX 3090 a way to run 70B?
Two 24 GB cards give 48 GB of VRAM, enough for Llama 3.3 70B at Q4_K_M at 8K context. This tool models one card, so use the "What do I need?" tab to see the 43 GB requirement and compare it with your total.
How accurate is the speed estimate for the 4090?
It is a ceiling from memory bandwidth, so real programs usually land somewhat below it. The relative comparison is reliable: an 8B model is roughly twice as fast as a 14B model on the same card.