Hebrew has been something of an afterthought in the LLM race. Most tokenizers are built for English and other morphologically simple languages, which means they butcher Hebrew’s system of roots, prefixes, suffixes, and infixes. A single Hebrew root can spawn dozens of valid word forms, and models trained on byte-pair encoding often treat those forms as unrelated tokens. The result: fluent-sounding Hebrew that falls apart under scrutiny.
A new open leaderboard aims to fix that. Sponsored by Israel’s DDR&D IMOD and the National Program for NLP in Hebrew and Arabic, with collaboration from DICTA and Webiks, the project evaluates models on four tasks: question answering (using the HeQ test set), sentiment classification, a Hebrew translation of the Winograd Schema Challenge, and English-Hebrew translation drawn from the NeuLabs-TedTalks corpus. All tests run in a few-shot format, so models can’t coast on fine-tuning tricks. Prof. Reut Tsarfaty of Bar-Ilan University provided scientific consultation.
The technical stack is deliberately simple. The leaderboard borrows its design from Hugging Face’s Open LLM Leaderboard, deploys submitted models automatically through Inference Endpoints, and runs evaluations via the lighteval library. The team says implementation was straightforward — the environment setup was the main lift, and the rest of the code ran smoothly.
Whether this moves the needle depends on who actually submits models. Hebrew-specific LLMs remain scarce, and the leaderboard’s value will hinge on whether Israeli startups and academic labs treat it as a standard rather than a curiosity. The Winograd component is particularly interesting: pronoun resolution in Hebrew forces models to contend with gender and number agreement in ways English never does. If nothing else, the leaderboard gives researchers a shared yardstick for a language that has largely been measured with borrowed rulers.
