CPU AI: Local LLM Speed Tested on Laptops

Local LLMs on laptops can be slow. Models with fewer than 2 billion parts run at 18-36 tokens/sec, which is fast enough for talking. Bigger models are much slower.
By Newsroom | Technology, AI |

Core Findings Emerge on Local LLM Viability Without Dedicated GPUs

Testing on Linux reveals that tokens per second (tok/s) emerges as the critical metric for assessing the actual usability of Large Language Models (LLMs) on CPU-only setups, particularly on less powerful hardware. Models under 2 billion parameters can achieve between 18-36 tok/s, rendering them responsive for daily interaction. Models in the 3-4 billion parameter range offer a workable, albeit slower, experience at 7-10 tok/s. Larger models, exceeding 7 billion parameters, dip to 3-4 tok/s, making them more suitable for asynchronous tasks where immediate response isn't paramount.

Olivia Attwood puts on a leggy display in a polka dot mini dress as she parties the night away before being left painfully hungover in more snaps from her influencer-packed Miami trip - 1

This performance envelope was established through experiments involving eight different LLMs on a laptop with an Intel i5 processor and 12GB of RAM. Specific models tested included Qwen 0.6B, TinyLlama 1.1B, Gemma 3 1B, Gemma 4 E2B, Granite 4 3B, Phi 4 Mini 3.8B, OpenHermes 7B, and Mistral 7B. The Phi 4 Mini, despite its slower pace, showed promise for tasks requiring reasoning, provided users can tolerate the increased latency.

Read More: Google AI Search Gives Bad Advice, Nvidia Driver Help Found

Olivia Attwood puts on a leggy display in a polka dot mini dress as she parties the night away before being left painfully hungover in more snaps from her influencer-packed Miami trip - 2

Quantization and Model Size Dictate CPU Performance

Strategies for Enhanced CPU Inference

The capacity to run LLMs locally without relying on powerful GPUs is heavily influenced by two key factors: model size and the implementation of quantization techniques. Quantization effectively compresses AI models, reducing their memory footprint and computational demands, thereby making them feasible for CPUs.

Olivia Attwood puts on a leggy display in a polka dot mini dress as she parties the night away before being left painfully hungover in more snaps from her influencer-packed Miami trip - 3

Quantization is presented as essential for CPU-only systems, enabling the running of larger models on consumer hardware with a minimal impact on performance. Techniques like 4-bit and 8-bit quantization are highlighted, alongside methods such as BitsAndBytes and PyTorch Ahead-of-Time (TorchAO) optimization, which facilitate efficient inference through model compression.

Olivia Attwood puts on a leggy display in a polka dot mini dress as she parties the night away before being left painfully hungover in more snaps from her influencer-packed Miami trip - 4

Model Performance Tiers

  • 1B–2B Parameter Models: Offer the best balance of speed and responsiveness for typical daily use. Examples include TinyLlama and Gemma.

  • 3B–4B Parameter Models: Provide a functional experience, though with noticeable slowdowns. Phi 4 Mini is noted for its reasoning capabilities within this tier.

  • 7B+ Parameter Models: Performance drops significantly, positioning them for background or less time-sensitive operations. Mistral 7B and OpenHermes 7B fall into this category.

Broader Context: Local AI and Its Implications

The drive towards running LLMs locally, independent of cloud infrastructure, is motivated by several factors. Primary among these are the elimination of ongoing API costs and the significant advantage of maintaining data privacy, as user information never leaves the local machine. This shift also presents a valuable learning opportunity for individuals seeking hands-on experience with AI model deployment.

Read More: Codex AI coding now on ChatGPT mobile apps for developers

Tooling and Ecosystem

Various tools and frameworks are emerging to facilitate this local AI movement. Ollama is mentioned as a straightforward method for getting models running on personal machines, with a command like ollama run mistral-small simplifying the process. Projects like localllm, integrated within the Google Cloud ecosystem, aim to enhance developer productivity by allowing the use of LLMs directly within cloud workstations.

Hardware Considerations and Limitations

While the focus is on CPU performance, attempts to leverage integrated graphics (iGPUs) on platforms like AMD systems have encountered difficulties. Specific issues arise from the mismatch between how tools like Ollama detect VRAM and how ROCm (AMD's compute stack) allocates memory on newer Linux kernels. This can result in iGPUs not being recognized or utilized effectively for running models, even when system memory is available. The challenge stems from ROCm's use of unified memory architecture and its interaction with the Graphics Translation Table (GTT), which can be incompatible with certain GPU detection logic.

Read More: New YouTube Tool Helps Find Specific Comments Fast

Frequently Asked Questions

Q: How fast do local AI models need to be on a laptop CPU?
Local AI models need to run at 18-36 tokens per second to be useful for daily tasks. This speed is possible with models under 2 billion parts.
Q: What AI models work well on a laptop CPU without a graphics card?
Models with 1-2 billion parts like TinyLlama and Gemma work well. Models with 3-4 billion parts like Phi 4 Mini are okay but slower. Larger models are too slow.
Q: How can AI models run better on a laptop CPU?
Using techniques like 4-bit or 8-bit quantization helps shrink AI models. This makes them use less power and run faster on CPUs.
Q: Why do people want to run AI models on their own computers?
Running AI models locally saves money on cloud fees and keeps your data private. It also helps people learn how to use AI.
Q: What problems happen when trying to use a laptop's graphics chip for AI?
Sometimes, tools like Ollama don't see the graphics chip correctly on Linux. This is because of how AMD's ROCm system handles memory, making the chip not work for AI.