72% of Phones Can Now Run Llama 3.2 Locally—Even the 5-Year-Old Ones

The idea that generative AI needs the latest $1,000 phone is officially dead. Arm’s latest software push, pairing its KleidiAI acceleration library with the upcoming ExecuTorch 0.7 beta, puts LLMs on a staggering 3 billion existing devices. We’re not just talking about this year’s flagships. The key is an old CPU instruction called SDOT (Signed Dot Product), which has been baked into Arm chips since 2015. That means 72% of the Arm-based devices in pockets today—including phones that are three, four, or even five years old—can run models like Llama 3.2 1B with practical performance. Developers get this instantly. By embedding KleidiAI directly into ExecuTorch and XNNPack, the optimization becomes the default. No custom kernel tuning. No assembly code. The performance isn’t theoretical, either. On a Galaxy S24+, this combo already pushes over 350 tokens per second during prefill and over 40 tokens per second during decode. But the real story is on the low end. Arm explicitly calls out the Raspberry Pi 5 as a viable target, and on older phones, the decode speed often surpasses average human reading speed. That’s the threshold where an app doesn’t just technically function—it feels responsive.

The use cases shift from gimmicky chatbots to genuinely private infrastructure. A local LLM running entirely on-device, paired with speech-to-text and text-to-speech, becomes a fully offline smart assistant. Think about that: no cloud costs, no latency, and zero privacy concerns because your data never leaves your handset. As the blog post notes, such a device could “seamlessly interact with your connected devices.” The broader implication is a power shift away from hyperscalers. If a capable model can summarize your 50 unread messages (roughly 600 tokens) on a budget phone from 2020, the economic argument for paying a cloud API per token starts to crack.

This is a quiet but profound infrastructure play. By defaulting KleidiAI into ExecuTorch—Meta’s on-device inference runtime—Arm is essentially making CPU-based AI a turn-key feature for Android and cross-platform developers. The I8MM extension on newer v8.6 chips gets all the headlines, but the SDOT work is what democratizes the experience. It’s the difference between demonstrating a cool tech demo on a test bench and shipping a feature to the next billion users. The upcoming 0.7 beta isn’t just a performance bump; it’s a distribution mechanism for an AI experience that can actually scale.