Liquid AI's Pipette Exposes Why Two 350M Models Keep 78% vs 34% Speed on Phones

Model cards are lying to you. Not maliciously — but the throughput numbers you see on a release page come from server-class hardware running full precision. The moment that same model hits a phone, everything changes. Liquid AI just open-sourced Pipette, a benchmarking platform built with Artificial Analysis as an independent validator, and the early data makes the gap impossible to ignore.

Pipette’s core insight is that on-device performance isn’t a property of the model. It’s a property of the deployment: model plus quantization plus runtime plus device. The launch dataset covers five metrics across more than 1,000 configurations — 30+ models, multiple quantization formats, llama.cpp builds for macOS, iOS, Windows, and Android, with context lengths from 256 to 8,192 tokens. Initial verified results come from an M5 Max MacBook Pro, iPhone 17 Pro, and Galaxy S26 Ultra. AMD Ryzen AI Max+ 395 numbers are listed as coming soon.

The headline comparison is brutal. Two Granite models, both 350M parameters, both at Q4_K_M on the same Galaxy S26 Ultra. At 4,096 input tokens, one retains 78.4% of its decode throughput relative to the 256-token baseline. The other keeps just 33.8%. Same parameter count, same quantization, same phone — completely different context-scaling behavior. Meanwhile, LFM2.5-8B-A1B decodes 2.4x faster than Qwen3.5-4B on the same device, but still eats 5.29 GiB of memory because sparse activation saves compute, not weights. And on iPhone 17 Pro, MiniCPM5-1B finishes a 2,048-in/256-out workload 15.8% faster than LFM2.5-1.2B, yet scores 9 points lower on MATH-500. Speed and quality don’t track together.

Pipette ships as Apache 2.0 infrastructure: management tooling, client libraries, scoring code, a public results dataset, a hosted dashboard, and native iOS and Android benchmark apps. Nothing is waitlisted. Community-submitted results are in beta. The quality scores shown alongside phone throughput come from llama.cpp runs on NVIDIA H100 systems — matched by model and quantization, not measured on the phone itself. That’s a reasonable proxy, but worth remembering when you see a MATH-500 score next to an iPhone latency number. The real test will be whether hardware vendors and model shops adopt this as a neutral yardstick, or whether we end up with another benchmark that everyone cites and no one trusts for procurement decisions.