Cosmovex/tools

Run a local LLM with no graphics card: what 16 GB of RAM can handle

CPU only, 16 GB RAM with Local AI Fit

This page opens with a CPU-only machine with 16 GB of dual-channel DDR4-3200 RAM at 8K context. Without a graphics card, generation speed is set by how fast RAM can be read, about 51 GB/s here. The list shows what fits and what speed to expect.

  1. Set the RAM size and type to match your machine.
  2. Read which models fit and the speed estimated for each.
  3. Tap Llama 3.1 8B to see the memory it takes and why it is slow.
  4. Switch the RAM type to DDR5-5600 to see how much faster memory helps.
  5. Pick a smaller quantisation or model if the speed is too low.
run llm on cpu only local llm without gpullm 16gb ram no gpuollama cpu onlycpu inference speed
Open Local AI Fit Free to start · no account

What to know

After the operating system takes about 4 GB, a 16 GB machine has about 12 GB for a model. Llama 3.2 3B at Q8_0 fits and runs near 10 tokens per second by this estimate, which is fine for chat. Llama 3.2 1B is near 25. Llama 3.1 8B at Q8_0 fits at about 9.6 GB but runs near 4 tokens per second, because each token reads the whole model from RAM at about 51 GB/s.

Memory speed is the lever. Two sticks of DDR5-5600 read at about 90 GB/s, so the same 8B model is roughly 1.75 times faster than on DDR4-3200. Running two sticks matters too: a single stick halves the bandwidth. The CPU itself matters less than people expect for generation, though it matters for processing long prompts.

Mixture-of-experts models change the picture on CPU. gpt-oss 20B reads only about 3.6 billion parameters per token, so at Q3_K_M (about 10.6 GB, which fits in 16 GB RAM) the estimate is about 13 tokens per second, faster than an 8B dense model. Quality at Q3 is reduced, so treat that as an experiment rather than a recommendation.

Updated · Cosmovex

Questions

Can I run an LLM without a GPU?

Yes. A CPU can run any model that fits in RAM; it is just slower. Small models of 1B to 3B parameters are comfortable and 7B to 8B models run at a few tokens per second.

How much RAM do I need for an 8B model on CPU?

About 6 GB for Q4_K_M and about 10 GB for Q8_0 including the cache at 8K context, plus about 4 GB for the system. A 16 GB machine is comfortable at Q4_K_M and tight at Q8_0.

Will more CPU cores make it faster?

Only up to a point. Generation is limited by memory bandwidth, so beyond the cores needed to saturate it, more cores add little. Faster RAM and two populated memory channels help more.

Is it worth adding a small graphics card?

Often yes. An 8 GB card holds an 8B model at Q5_K_M entirely and runs it near 30 tokens per second by this estimate, about eight times faster than the CPU alone. See the RTX 4060 8GB page for the numbers.