Cosmovex/tools

Run Qwen3 32B locally: the memory it needs and which cards hold it

Run Qwen3 32B with Local AI Fit

This page opens on the reverse tab for Qwen3 32B at Q4_K_M and 8K context. It shows the memory required, the graphics cards and Macs that hold it entirely, and how the number grows with context length. DeepSeek R1 Distill 32B shares the same shape.

  1. Read the memory required at the top of the result.
  2. Change the quantisation to Q5_K_M or Q6_K to see what extra quality costs.
  3. Raise the context to 32K and see how much of a 24 GB card the cache takes.
  4. Check the graphics cards and Macs that hold it.
  5. Switch to "Will it run?" to test your own hardware.
qwen3 32b vram requirements qwen3 32b hardwarerun qwen 32b locallyr1 distill 32b vram32b model 24gb vram
Open Local AI Fit Free to start · no account

What to know

Qwen3 32B has about 32.8 billion parameters. At Q4_K_M the weights are about 18.5 GB, the cache at 8K context about 2 GB, and the runtime 0.7 GB, so it needs about 21 GB. A 24 GB card has about 22.6 GB usable and holds it, with little room left for a longer context; at 32K the cache alone is about 8 GB, and the model no longer fits on a 24 GB card.

A 32 GB card changes this. The RTX 5090 holds Q6_K (about 27.7 GB at 8K context) or Q4_K_M with a long context. On the Mac side, a 32 GB machine has about 24 GB usable by default, so it holds Q4_K_M at short contexts; 48 GB or more is comfortable. Estimated speeds are about 33 tokens per second on a 4090 and about 27 on an M3 Ultra.

It is a dense model, so every token reads all of it. That makes memory bandwidth the main factor, and any share that spills into system RAM is expensive: a 12 GB card with 32 GB of RAM manages about 5 tokens per second by this estimate. With only 12 to 16 GB of VRAM, the 14B models or the mixture-of-experts options will feel much better.

Updated · Cosmovex

Questions

Can a 24 GB card run a 32B model?

Yes, at Q4_K_M with a context up to roughly 8K to 16K. Longer contexts need more cache memory and push the model into system RAM or force a lower-quality quantisation.

Does R1 Distill 32B need the same memory?

Yes. DeepSeek R1 Distill Qwen 32B uses the same architecture as the Qwen2.5 32B family, so the weights and cache are the same size as for Qwen3 32B at the same quantisation.

What is the benefit of Q6_K over Q4_K_M?

A small quality improvement for about 35% more memory: roughly 25 GB against 18.5 GB of weights. On a 32 GB card it is worth it; on a 24 GB card it does not fit.

Is Qwen3 30B-A3B a better choice than the 32B?

For speed, yes: it is a mixture-of-experts model that reads about 3.3 billion parameters per token, so it is several times faster. It needs somewhat less memory than the 32B at the same quantisation, and the quality of the two differs by task.