GPT-4o still beats local LLMs in Filipino, but a 2-3% fine-tuning gain keeps the underdogs in the race

The new FilBench evaluation paints a clear but nuanced picture of AI’s relationship with Philippine languages. After testing over 20 large language models on a suite of 12 tasks—ranging from Cebuano readability to Filipino-centric values—the researchers found a predictable winner and a surprising reason for optimism.

“The best SEA-specific model is still outperformed by closed-source LLMs like GPT-4o,” the team notes. That’s not a shock. What is interesting is the efficiency curve. Regional models like SEA-LION and SeaLLM punch well above their weight class, achieving the highest scores for their parameter count. The secret sauce? It’s not just about scale. The paper shows a concrete performance uplift of 2-3% when you continuously fine-tune a base model on Southeast Asian instruction data. That’s a specific, measurable argument against the idea that only trillion-parameter models from Silicon Valley matter.

The real carnage happens in translation. The Generation category, which includes English-to-Filipino and Cebuano-to-English tasks, is where models consistently faceplant. The failure modes aren’t subtle—they’re spectacular. Models ignore the translation instruction entirely, produce comically verbose output, or simply hallucinate text in a completely different language. If you were hoping to use a cheap LLM for production-grade Filipino translation, the message from FilBench is blunt: the tools aren’t ready, and the errors aren’t graceful.

What makes FilBench compelling is its deliberate refusal to use translated content for most tasks. The benchmark is built on native datasets like KALAHI for values and the Cebuano Readability Corpus, which tests actual comprehension rather than a model’s ability to parrot translated English. The authors didn’t just throw together a quiz—they based the task categories on a historical survey of NLP research in Philippine languages from 2006 to early 2024. That creates a benchmark that measures what researchers actually care about, not just what’s easy to test. The entire project is now available as a community task in Lighteval, which means it’s trivially reproducible. For a country where compute budgets are tight, that openness is arguably as important as the leaderboard itself.