Google's CodeGemma 7B beats all but DeepSeek on coding benchmarks

Google just dropped CodeGemma, a family of open-access code-specialist models built on the Gemma 2B and 7B checkpoints. The models were trained on an additional 500 billion tokens of English language data, mathematics, and code to sharpen their logical and mathematical reasoning. Three versions are available: a 2B base model focused exclusively on code infilling for fast completion in latency-sensitive settings, a 7B base model trained with an 80/20 mix of code infilling and natural language, and a 7B instruct variant fine-tuned for conversational use around programming and math.

The 7B base model is the headline performer. On HumanEval, the standard Python benchmark, CodeGemma-7B outperforms similarly-sized 7B models except DeepSeek-Coder-7B. That pattern holds across other languages tested via MultiPL-E, including Java, JavaScript, and C++. It also claims the top spot among 7B models on GSM8K, a mathematical reasoning benchmark. The instruct version improves further on the most popular languages for both HumanEval and MBPP.

For code completion, the 2B and 7B base models use a fill-in-the-middle approach with dedicated tokens — <|fim_prefix|>, <|fim_suffix|>, and <|fim_middle|> — plus a <|file_separator|> token for multi-file contexts. The instruct model follows the same conversation structure as Gemma’s instruction-tuned releases, using the standard <start_of_turn> format that transformers’ chat template handles automatically.

Hugging Face collaborated directly with Google on the release, and the models are ready to use on the Hub with full transformers integration as of release 4.39. That means safetensors format, bitsandbytes 4-bit quantization, PEFT fine-tuning, Flash Attention 2, and torch.compile() compatibility all work out of the box. There are also GGUF files for community quantization, Google Cloud integration, and Inference Endpoints support. The fact that CodeGemma-7B sits just behind DeepSeek-Coder-7B on HumanEval while beating it on GSM8K suggests Google is positioning this as the more well-rounded option for teams that need both code generation and mathematical reasoning without running multiple specialized models.