How a 3B open model taught itself to rethink math—no human data needed

The release of DeepSeek-R1 sent a jolt through the AI world. Here was an open model going toe-to-toe with OpenAI’s o1 on complex reasoning, and its creators published the playbook. At the heart of that playbook is an ‘aha moment’—a phase during pure reinforcement learning where the model learns, without a single human example, to allocate more thinking time and reevaluate its initial approach. That’s the spark a new tutorial from Inblix aims to replicate on a much smaller scale.

This post walks through reproducing that emergent self-verification using Group Relative Policy Optimization (GRPO) and the Countdown Game. The setup is surprisingly accessible: a 4x NVIDIA H100 GPU server, the Qwen/Qwen2.5-3B-Instruct model, and a dataset of number puzzles where the goal is to hit a target using basic arithmetic. The tutorial leans on the TRL library, DeepSpeed, and vLLM for distributed training, mirroring the efficiency gains that make GRPO attractive in the first place—ditching the clunky value function model from traditional PPO and instead normalizing rewards within groups of generated outputs.

GRPO works in a straightforward loop: the current policy generates multiple answers per prompt, each gets scored by a rule-based reward function, and the average reward becomes a baseline. Solutions better than the group average get a positive advantage signal; worse ones get a negative one. The policy then optimizes against that advantage, with a KL-divergence penalty preventing it from drifting into gibberish. As the blog notes, this approach has already been adopted by the Qwen team, signaling it’s not just an academic curiosity.

The real intrigue is in the training observations. Following early work by Jiayi Pan, the tutorial suggests that even a 3B parameter instruction-tuned model can develop search-like behaviors when given nothing but outcome-based rewards. The model starts to check its own work—not because anyone told it to, but because the math of group-relative advantage rewards that behavior. It’s a controlled, reproducible version of the emergent reasoning that made DeepSeek-R1’s paper worth reading in the first place. Whether this scales as smoothly to the frontier models without the multi-stage training DeepSeek later added remains an open question, but for teams looking to bootstrap reasoning in smaller models, the recipe is now public and well-documented.