Cosmovex/tools

RTX 5090 32GB for local LLMs: 32B at Q6, 70B at Q4 and gpt-oss 120B

RTX 5090 32GB with Local AI Fit

This page opens with an RTX 5090 32GB and 64 GB of DDR5-6400 RAM at 8K context. The 32 GB of memory and 1,792 GB/s of bandwidth change which models are comfortable. It shows what fits entirely and what still needs system RAM.

  1. Confirm the card and the RAM at the top.
  2. Read which models run entirely on the card.
  3. Tap Qwen3 32B to see the extra headroom that 32 GB gives over 24 GB.
  4. Tap Llama 3.3 70B to see the share that goes into system RAM.
  5. Compare with the 24 GB page by opening the RTX 4090 variant.
rtx 5090 llm 5090 32gb local llmcan 5090 run 70brtx 5090 vram llmgpt-oss 120b 5090
Open Local AI Fit Free to start · no account

What to know

About 30 GB is usable on a 32 GB card. Qwen3 32B moves from Q4_K_M on a 24 GB card to Q6_K here (about 27.7 GB at 8K context, near 43 tokens per second by this estimate), a visible quality improvement. Mixtral 8x7B at Q4_K_M, about 28 GB, fits entirely and is estimated above 100 tokens per second because it reads only about 12.9 billion parameters per token.

Llama 3.3 70B at Q4_K_M needs about 43 GB. About 70% of it sits on the card and the rest in RAM, and the estimate is about 4.5 tokens per second: slow but workable for non-interactive use. gpt-oss 120B at Q4_K_M needs about 67 GB; only about 5.1 billion parameters are read per token, so the estimate stays near 26 tokens per second even though more than half of it lives in RAM. For mixture-of-experts models the memory capacity matters more than the speed of the part in RAM.

Two things limit the 5090 for this use. Its memory is bigger than any consumer card before it but still smaller than a 70B model at Q4 needs, and the model has to be read from system RAM at 90 to 100 GB/s once it spills. If 70B models at conversational speed are the goal, check the Mac and multi-card options in "What do I need?".

Updated · Cosmovex

Questions

Does 32 GB of VRAM run Llama 70B?

Not entirely. Llama 3.3 70B at Q4_K_M needs about 43 GB at 8K context. With 64 GB of RAM it runs split at roughly 4 to 5 tokens per second by this estimate. At Q2_K it needs about 28 GB and would fit on the card, but the quality loss is large.

Why is gpt-oss 120B fast when most of it is in RAM?

It is a mixture-of-experts model: only about 5.1 billion of its 117 billion parameters are used for each token. The weights still have to be stored somewhere, which is why it needs about 67 GB, but the amount read per token is small.

What does the extra 8 GB over a 4090 buy?

Mainly quality and context. A 32B model moves from Q4_K_M to Q6_K, and long contexts that overflowed a 24 GB card fit. Models above 32B still need system RAM.

Is system RAM speed important with a 5090?

Only for models that spill. DDR5-6400 reads at about 102 GB/s against 1,792 GB/s on the card, so any part of the model in RAM dominates the time per token. Keep the model on the card when you can.