Google's Gemma 2 drops: A 27B open model that beats Llama 3 70B

Google just released Gemma 2, and the numbers are frankly uncomfortable for anyone who bought into the bigger-is-better narrative. The 27B instruction-tuned variant outperforms Meta’s Llama 3 70B on the LMSYS Chatbot Arena leaderboard — that’s a model less than half the size beating a competitor with nearly three times the parameters. The 9B version isn’t just competitive; it’s dominating its weight class entirely.

The technical recipe is where things get interesting. Gemma 2 alternates sliding window attention (4096 tokens) on every other layer with full quadratic global attention across the 8192-token context, a hybrid approach that keeps memory costs down while preserving long-context quality. Soft-capping keeps attention logits from exploding — capped at 50.0 for attention, 30.0 for final logits — and Google used knowledge distillation from a larger teacher model to pre-train the 9B variant. The 27B was trained from scratch on 13 trillion tokens. Both ran on Google’s TPU infrastructure: v5p for the 27B, v4 for the 9B.

What Google isn’t saying is almost as telling. The training data mix, fine-tuning datasets, and SFT/RLHF hyperparameters remain undisclosed. That’s consistent with how Google has handled previous open releases — permissive license, limited transparency on the details that would let researchers fully reproduce the results. The license itself allows commercial use, fine-tuning, and derivative works, so the ecosystem can build on this foundation without legal friction.

For developers, the practical implications are immediate. You can fine-tune Gemma 2 with TRL on Hugging Face, deploy it through Google Cloud Vertex AI, or run inference via Inference Endpoints. The 9B model’s use of distillation rather than pure scale suggests we’re entering an era where training efficiency — not parameter count — is the real competitive moat. One caveat: soft-capping is incompatible with Flash Attention and SDPA during fine-tuning, so stable training runs require eager attention. Minor sacrifice, substantial performance payoff.