ggml packs PyTorch power into 1MB: here's why devs are switching

The library that quietly powers llama.cpp and Ollama is worth a close look for any developer who’s ever winced at a PyTorch Docker image. ggml — the tensor library by Georgi Gerganov — compiles to a binary under 1 MB. Compare that to the hundreds of megabytes PyTorch typically demands, and you immediately understand the appeal for on-device inference.

The trade-off is control. ggml isn’t a drop-in PyTorch replacement and it doesn’t pretend to be. The core API surfaces low-level concepts like ggml_context (a container for tensors and graphs), ggml_cgraph (the compute graph itself), and a backend system that spans CPU, CUDA, Metal, and Vulkan. Building even a simple matrix multiplication means manually allocating a context, creating tensors, building the graph, and wiring up a backend scheduler — steps PyTorch hides behind torch.matmul.

That verbosity buys predictability. Because ggml owns the entire memory lifecycle through its ggml_gallocr graph allocator and ggml_backend_buffer abstractions, there’s no garbage-collector pause and no framework overhead inflating RAM usage. The library also bakes in quantized tensor support, treating compression as a first-class operation rather than an afterthought — a design choice that matters enormously when you’re squeezing a 7B-parameter model onto a phone.

It’s not all smooth sailing. The project is under active development with breaking changes expected, and not every operation is available on every backend. A matmul that runs fine on CPU may silently fail on CUDA. This is a library for developers comfortable reading source code — the entire core spans fewer than five files — and the documentation assumes you’re willing to get your hands dirty. That minimalism is the point. When something breaks, you can actually trace why.