NeurIPS 2025 bets $16K that benchmarks, not loss curves, can judge baby LLMs
Curated by the Inblix editorial team
The blank stare of a loss curve is a familiar frustration for anyone who has trained a large language model. In the early stages, when a model has only seen around 200 billion tokens, standard benchmarks are basically useless. They don’t provide a meaningful signal about whether a particular architecture or data mix is actually working. It’s a guessing game that wastes enormous compute. The new NeurIPS 2025 E2LM Competition wants to kill that guessing game by crowdsourcing evaluation benchmarks that can spot a clever model early, specifically within the scientific knowledge domain.
Hosted on Hugging Face and powered by the lm-evaluation-harness library, the competition provides a refreshingly low barrier to entry. Organizers are releasing 0.5B, 1B, and 3B parameter model checkpoints from the first 200 billion tokens of training. Participants can run everything on a free-tier Google Colab GPU, which democratizes a challenge that would normally be locked inside well-funded AI labs. A starting kit with notebooks is provided to get people moving.
Submissions aren’t judged on a single vector. The scoring is a weighted composite of three distinct criteria. Signal quality, which measures how well a benchmark discriminates between model stages, dominates at a 0.5 weighting. Compliance with established scientific knowledge is close behind at 0.4. The remaining 0.1 goes to ranking consistency. Crucially, participants can only calculate the signal quality score locally. The other scores rely on hidden model checkpoints—trained up to 1 trillion tokens and on web-only data—to prevent participants from gaming the system with overfit solutions. The overall score is computed automatically upon submission.
The prize pool is a pragmatic mix of open and targeted awards. First place takes $6,000, second $4,000, and third $2,000. There’s also a specific nod to academia with two $2,000 student awards. It’s a smart structure that acknowledges this problem isn’t just about raw cash—it’s about building a standard that could quietly reshape how AI researchers decide which experiments to kill and which to scale, saving millions in wasted compute in the process.
💡 Key Takeaways
- Current benchmarks fail to provide discriminative signals during the first ~200B tokens of LLM training, forcing researchers to rely on vague metrics like loss curves.
- The competition uses a weighted scoring system that heavily penalizes scientific inaccuracy (0.4 weight) and rewards clear signal quality (0.5 weight) to prevent superficial solutions.
- Hidden model checkpoints are used during evaluation to prevent participants from overfitting their benchmarks to the publicly released early-training models.
Keep reading: See related articles below for more coverage on this topic.
Get smarter about AI
The sharpest AI news, curated daily. Delivered free to your inbox.