Mistral 7B now runs on Macs in under 4GB memory via Apple's Core ML update

Apple’s WWDC 24 Core ML updates make running a 7-billion-parameter model on consumer hardware surprisingly practical. The key stat: Mistral 7B now executes in under 4GB of memory on a Mac, a threshold that would have seemed fanciful even a year ago for a model of this size. The secret is a pair of new features Apple quietly shipped in iOS 18 and macOS Sequoia.

The first is MLTensor, a new Swift type that finally gives Apple developers something resembling the tensor manipulation convenience Python developers have enjoyed for years with NumPy and PyTorch. Before this, working with Core ML’s MLMultiArray or MLShapedArray meant writing custom code just to access underlying storage through opaque pointers. Operations as basic as softmax required hand-rolled Accelerate code. Now those primitives ship out of the box, which matters enormously for anyone porting transformer architectures to Swift.

The second is stateful buffers. Core ML models were previously stateless functions—input goes in, output comes out, and any persistent state had to be passed back and forth as additional I/O pairs. For LLMs, which run hundreds of iterations to generate a single sequence, that round-tripping of large state data creates serious overhead. The bottleneck isn’t compute; it’s memory bandwidth. Keeping state resident on-device eliminates that churn entirely.

Hugging Face has already adapted swift-transformers to leverage both features, with a preview branch available on GitHub and converted Mistral 7B models hosted for download. Apple showcased the approach during its ‘Deploy machine learning and AI models on-device with Core ML’ session, and the broader implication is hard to miss: on-device LLMs are no longer a research curiosity. They’re becoming a developer target. The same vertical integration that powers Apple Intelligence is now open to anyone shipping apps on Apple platforms.