Run Llama 70B locally with Local AI Fit
This page opens on the reverse tab for Llama 3.3 70B at Q4_K_M and 8K context. It shows the memory required, which graphics cards and Mac configurations hold the model entirely, and how the number changes with quantisation and context.
- Read the memory required at the top of the result.
- Change the quantisation to see the memory at Q8, Q5 and Q3.
- Change the context length to see how much the cache adds.
- Check the list of graphics cards; an empty list means no single card in the table holds it.
- Read the Mac configurations, which are the smallest single machines that hold it.
What to know
The weights of a 70.6 billion parameter model at Q4_K_M, about 4.85 bits per weight, are about 39.9 GB. The context cache adds about 2.5 GB at 8K tokens and about 10 GB at 32K tokens, and the runtime another 0.7 GB, so the total is about 43 GB at 8K and about 51 GB at 32K. At Q8_0 the weights alone are about 70 GB; at Q3_K_M they are about 32 GB.
No single consumer graphics card in the table holds 43 GB. Options are two 24 GB cards (48 GB total, tight with a long context), a workstation card with 48 GB or more, a Mac with at least 64 GB of unified memory (about 48 GB of it usable by default), or splitting the model into system RAM at a large speed cost. Estimates for a 64 GB Mac range from about 6 tokens per second on an M1 Max or M2 Max to about 12 on an M2 Ultra.
The R1 Distill 70B models share the Llama 70B architecture, so the same numbers apply to them. Reasoning models produce long chains of thought, which makes both the speed and the context length matter more than for a short chat.
Updated · Cosmovex
Questions
How much VRAM do I need for Llama 3.3 70B?
About 43 GB at Q4_K_M with an 8K context, about 51 GB with a 32K context, about 32 GB of weights at Q3_K_M, and about 70 GB of weights at Q8_0. Anything less means part of the model runs from system RAM, which is much slower.
Can I run Llama 70B with only system RAM?
Yes, if you have enough: about 48 GB of RAM for Q4_K_M including the system share. On dual-channel DDR5-5600 the estimate is about 1 token per second, so it works for experiments and not for conversation.
Is a Mac a sensible way to run a 70B model?
It is the cheapest way to hold one in fast memory on a single machine. A 64 GB or 96 GB Mac with a Max or Ultra chip holds Q4_K_M and runs at roughly 6 to 12 tokens per second by this estimate, which is usable for chat.
What does Q4_K_M lose compared with the original?
Some accuracy, usually small for a model this large. Larger models tolerate quantisation better than small ones, which is why Q4 on a 70B model is often better than Q8 on a smaller model of similar file size.