Apple's Neural Engine is 12x More Power-Efficient Than CPU for On-Device OCR

Running state-of-the-art AI models on Apple devices just got a clearer playbook. The team behind dots.ocr, a model combining a 1.2B-parameter vision encoder and a Qwen2.5-1.5B language backbone, has detailed their arduous journey of converting a PyTorch model to run on Apple’s Neural Engine. The payoff is significant: their testing found the Neural Engine to be 12x more power efficient than the CPU and 4x more efficient than the GPU.

But that efficiency comes with a steep entry price. The Neural Engine is locked behind Core ML, Apple’s closed-source machine learning framework. As the developers bluntly state, converting a model from PyTorch to Core ML without prior experience is “arduous for developers” due to its many “sharp edges.” To bypass some of this friction, they also utilized MLX, Apple’s more modern and flexible framework that targets the GPU, creating a hybrid approach: Core ML runs the vision encoder, and MLX handles the language model backbone.

The conversion process, detailed in a three-part series, is a narrative of tactical simplification and bug hunting. The team first gutted the vision encoder to a minimal viable state, stripping out multiple attention implementations to focus solely on Apple’s scaled_dot_product_attention operator from iOS 18. Their first conversion attempt immediately crashed with a dtype mismatch error in the rotary position embedding layer. The error log pointed to an int32 and fp32 conflict in a matrix multiplication operation, a common class of issue when translating execution graphs.

This meticulous trace of fixing errors “further and further down the execution graph” is framed as a generalizable approach for any developer looking to ship their own on-device models. It’s a sobering reminder that while the hardware is astonishingly efficient, the software toolchain remains a gauntlet of precision mismatches and graph capture quirks that require manual surgery to navigate.