llama.cpp now juggles multiple models without crashing, just like Ollama

The llama.cpp project just shipped one of its most-requested features: native model management that lets a single server handle multiple LLMs, loading and unloading them on the fly. If you’ve ever envied Ollama’s ability to switch between models without restarting a server, you now have a lightweight, OpenAI-compatible alternative that doesn’t require a separate daemon.

The new router mode kicks in when you launch llama-server without specifying a model. It auto-discovers GGUF files from your llama.cpp cache or any directory you point it to with --models-dir. On the first request for a given model, it loads into memory. Subsequent calls hit the already-warm model. When you reach the --models-max limit (which defaults to 4), the least-recently-used model gets evicted. It’s not flashy, but it’s the kind of practical engineering that makes local AI development less tedious.

Under the hood, each model runs in its own process. That’s the key architectural decision here — if one model crashes, it doesn’t take the others down with it. The router handles request routing based on the model field in your API call, so you can literally A/B test two different quantizations or architectures from the same endpoint. All models inherit settings like context size and GPU layers from the router, but you can override per-model settings with a presets config file.

The API is dead simple. Chat completions work the same as always, just add the model name. There are new endpoints for listing available models, manually loading a model, and unloading one to free up VRAM. The built-in web UI even got a model selector dropdown. If you’re running multi-tenant setups or just flipping between a coding model and a creative writing model during development, this eliminates the friction of restarting servers.