Gemma 3's 27B model punches into top 10 of Chatbot Arena, beating Gemini 1.5 Pro

Google just dropped Gemma 3, and the numbers are genuinely surprising. The new open-weight model family runs from a tiny 1B parameter version up to a 27B powerhouse, but here’s the headline: the 4B instruction-tuned model outperforms last generation’s Gemma 2 27B. That’s a staggering efficiency leap in a single generation. The top-spec 27B model goes further, beating Google’s own Gemini 1.5 Pro across benchmarks and landing an Elo score of 1339 on the LMSys Chatbot Arena—good enough for a top-10 spot alongside closed-source giants. It’s a clear signal that the open model gap is collapsing fast.

Beyond raw performance, the feature set is what makes this release feel like a Swiss Army knife. The 4B, 12B, and 27B variants are now multimodal, capable of chewing on both images and text thanks to a SigLIP vision encoder. They also support 140 languages and boast a 128K token context window, a massive jump from Gemma 2’s 8K. The team achieved that context scaling without training from scratch by cranking up RoPE base frequency to 1M and tuning attention layers to interleave 5 local layers for every 1 global layer. For images, they use an adaptive cropping algorithm called ‘pan and scan’ that lets the model zoom into details on non-square or high-res images, treating visual tokens with full bidirectional attention while text stays causal.

There are smart memory tricks here too. Developers can run the models as text-only LLMs without loading the vision encoder into memory, keeping the footprint lean when you don’t need image analysis. The tokenizer gets an overhaul as well, switching to a 262K-entry SentencePiece model from Gemini 2.0 that dramatically improves encoding for Chinese, Japanese, and Korean—though English and code token counts take a slight hit.

The 1B model remains text-only but still gets a context bump to 32K tokens, making it an intriguing option for lightweight, long-document tasks. With both base and instruction-tuned checkpoints available and tight Hugging Face integration, Google is clearly betting that accessibility drives adoption. The question now is how the community will stack this against DeepSeek and Llama variants—and whether that 27B performance holds up on messy, real-world tasks beyond the leaderboards.