Google's Gemma 4 packs GPT-rivaling power into a 4B-parameter model

A new open model family from Google is making a very specific kind of promise: frontier-level performance without the data center. The Gemma 4 release spans five sizes, but the headline number belongs to the 26-billion-parameter mixture-of-experts variant. With only 4 billion active parameters, it scores an estimated 1441 on the LMArena text benchmark. Its dense 31B sibling pushes that to 1452. Those aren’t just good-for-their-size numbers. They put these models in the same conversation as much larger proprietary systems, and they run on-device.

The architecture doesn’t chase every new trend. Google stripped out what it calls “complex or inconclusive features” like Altup and instead doubled down on practical efficiency. The spec sheet is dense but coherent: alternating local sliding-window and global attention layers, a shared key-value cache across later layers to cut redundant projections, and a dual RoPE configuration to handle long contexts. On smaller models, a second embedding table feeds a residual signal into every decoder layer — a trick called Per-Layer Embeddings that lets the model specialize without ballooning the parameter count.

Multimodality is where the lineup gets interesting. Most variants handle images and text via a vision encoder that now supports variable aspect ratios and configurable token budgets, so you can actually tune the speed-memory-quality tradeoff. The two smallest models add audio through a USM-style conformer. But the 12-billion-parameter Unified model chucks the separate encoders entirely. It projects raw image patches and audio waveforms straight into the language model’s embedding space. That’s a genuinely different approach, and it will either be the most flexible option or the hardest to fine-tune — likely both.

Licensing might be the quietest but most important detail. Apache 2.0 across the board. No bespoke restrictions, no acceptable-use policy hidden in a separate webpage. Combined with first-class support for transformers, llama.cpp, MLX, WebGPU, and Rust, the distribution strategy looks designed to make these models the default choice for anyone building locally. The models are so capable out of the box that the team at Hugging Face admits they struggled to find compelling fine-tuning examples. When a base model is already that strong, the burden shifts to application design. That’s a good problem for developers to have.