The Korean LLM scene just got a proper measuring stick. Upstage launched the Open Ko-LLM Leaderboard in September 2023, and in five months it’s already pulled in over 1,000 model submissions. For context, the original English Open LLM Leaderboard needed considerably longer to reach its current 4,000+ models. That pace says something about how much pent-up demand exists for Korean-language evaluation.
The design philosophy is deliberately different from most leaderboards. Instead of open test sets that models can inadvertently train on, Upstage built entirely new private datasets for Ko-ARC, Ko-HellaSwag, Ko-MMLU, Ko-Truthful QA, and Ko-CommonGEN V2. The company argues that keeping test sets private prevents contamination and creates a fairer comparison. It’s a tradeoff — closed benchmarks mean less utility for the broader research community, but they also mean scores actually mean something. Given how often leaderboard gaming makes headlines, the caution feels warranted.
Participation spans individual researchers, universities, and major Korean companies like KT, Lotte Information & Communication, Yanolja, and MegaStudy. That corporate interest signals Korean businesses are taking local-language model evaluation seriously, not just treating it as an academic exercise. The benchmarks themselves are notable for being translated and adapted specifically for Korean linguistic and cultural contexts rather than simply porting English tests.
Whether a private test set approach scales remains an open question. The English leaderboard’s open format enabled an entire ecosystem of derivative benchmarks and research. Upstage is betting that fairness beats openness for now. With 1,000 models in five months, the Korean AI community seems willing to take that bet too.
