Arabic LLMs finally face a real STEM test, and most fail hard

The press release for a new benchmark called 3LM opens with a diplomatic observation: Arabic LLMs have progressed, but existing tests don’t measure what matters for technical work. That’s the polite version. The blunt reality buried in the results is that even the strongest models are nowhere near reliable for Arabic-language STEM reasoning or coding.

3LM, which stands for علم (knowledge), doesn’t just recycle sentiment analysis or summarization tasks. It’s a three-headed challenge built from 865 real multiple-choice questions ripped from Egyptian textbooks and exams for grades 8-12, 1,744 synthetically generated high-difficulty reasoning problems, and Arabic translations of the standard HumanEval+ and MBPP+ code benchmarks. The point was to see if models can actually think in Arabic about physics, chemistry, and Python, not just parrot facts.

The numbers are humbling. Across 40 evaluated models, Qwen2.5-72B-Instruct topped the leaderboard for multiple-choice STEM questions, but its scores tell the real story: 71.8% on native questions and 67.0% on the synthetic set. That’s a D+ on material designed for teenagers. For generative STEM answers, Gemma-3-27B managed a 43.2% accuracy. That’s failure by any practical standard. The code results are just as damning. GPT-4o, the model used to translate the code prompts into Arabic in the first place, was the best at solving them, which raises uncomfortable questions about whether the Arabic component is even being tested or if the model is simply reverting to its English training.

What’s genuinely new here is the construction of the benchmark itself. The team used a dual OCR pipeline with Pix2Tex to salvage both Arabic text and LaTeX math formulas from PDFs, then had native-speaking STEM experts verify the questions. The synthetic set wasn’t just hallucinated; it was generated from summarized textbook chunks and filtered to remove ambiguous or image-dependent nonsense. For code, they took the unusual step of leaving all test suites in English so the only variable is whether the model understands the Arabic prompt. That’s a clean experiment.

But a benchmark also reveals the limits of the field. The best model can’t crack three-quarters of generative STEM problems. The code benchmark’s top performer is the same proprietary model that built the benchmark. If you’re an educator in Cairo or a developer in Riyadh hoping to deploy these tools, the takeaway isn’t which model won. It’s that no model is ready, and the gap between Arabic and English technical performance remains a chasm, not a crack.