BenCzechMark exposes 25 LLMs: most still mangle Czech grammar and culture

The best open-source large language models still stumble badly over Czech, and now we have the receipts. A new evaluation suite called BenCzechMark puts over 25 open-weight models through a brutal 50-task obstacle course, and the results aren’t pretty for anyone who assumed English fluency translated seamlessly into other languages.

The project, the first comprehensive benchmark specifically for Czech, doesn’t just test whether a model can parrot translated trivia. It digs into genuine linguistic understanding with 90% native, non-translated content. This matters. A model can fake its way through a translated Wikipedia article, but asking it to fill in the correct past-tense verb suffix in the ‘Agree’ task or spot a grammar error in a language learner’s essay requires a deeper syntactic grasp that most models clearly lack.

BenCzechMark organizes its torture test into nine categories, from Math Reasoning and Factual Knowledge to a particularly devious Czech Language Understanding section sourced from real Czech school exams. The fact that tasks like the reading comprehension benchmark SQAD 3.2 and the grammar tests come directly from Czech Wikipedia and actual CERMAT exams (the Czech state exams) means the benchmark isn’t some academic abstraction. It tests the kind of practical, culturally-specific knowledge a Czech student is expected to have. The inclusion of language modeling tasks using the Czech National Corpus — spanning dialects, spoken language, and historical texts — also measures how well a model captures the true texture and diversity of the language, not just textbook Czech.

There’s a smart methodological choice here too. To avoid the common headache of calibrating models that have wildly different biases, the suite uses AUROC for classification tasks. This ranks predictions rather than requiring a perfectly set threshold, making it a fairer fight when comparing models of different sizes and training pedigrees. It’s a detail that suggests the creators have spent serious time thinking about why cross-language model evaluation is often a broken mess. The leaderboard with over 25 models should serve as a reality check for anyone building Czech-language applications and assuming GPT-4-class performance in their niche.