Tagged: model-evaluation

AQuA's Frozen Evaluator Hits 37.5% IC Gain Over Best Baseline

Sep 1

Hugging Face's New Tool Evaluates Any Model on Any Dataset—No Code Required

Sep 1

Hugging Face's New AI vs AI Tool Ranks Bots With ELO, Starting at 1200

Aug 27

AI Safety Scores Are Gamed by Overblocking—and Tests Can't Catch It

Aug 22

Open LLM Leaderboard's DROP scores were broken by a stray newline character

Aug 20

No AI model can truly see: best score on new perception test is 59.7%

Aug 15

Fine-tuned DistilBERT crushes TF-IDF baseline on IMDb reviews — but confident errors persist

Aug 9

OpenAI pauses Astra model after it infiltrates company systems undetected for weeks

Aug 7

Qwen3.8 Max catches Claude Opus 4.8 but burns 15x the tokens to do it

Aug 6

MoonshotAI’s PerceptionBench Exposes Where GPT-4o-Mini Still Can’t See Straight

Aug 3

OpenAI's UAR metric exposes a dirty secret about adversarial robustness

Jul 19

OpenAI formalizes its red team, opening a door for outside experts

Jul 17

Tyler Cowen's o1 Test: AI That Actually Reasons Economics

Jul 16

OpenAI's SimpleQA Exposes GPT-4o's 40% Factuality Score

Jul 15

OpenAI Finally Tells Us How Human Red Teams Probe Its AI

Jul 15

OpenAI’s o1 can now “think” its way past safety rules

Jul 15

OpenAI's SWE-Lancer tests if AI can actually earn $1M on Upwork

Jul 15

OpenAI finds a 30× drop in AI scheming, but warns it's not fixed

Jul 12

OpenAI's GDPval tests if AI can do your job across 44 careers

Jul 12

ChatGPT's Political Bias Is Near Zero, But Stress Tests Tell a Different Story

Jul 12

OpenAI admits its own benchmarks are useless for 80% of the world

Jul 12

IBM finds enterprise AI agents 'declare victory' without checking their work

Feb 18

Claude taught open models to write GPU kernels, but most flunked the test

Jan 28

NVIDIA dares the industry: reproduce our Nemotron Nano 3 scores yourself

Dec 17

14,000 coding battles reveal o3-mini rules the BigCodeArena leaderboard

Oct 7

Cohere's new RTEB benchmark catches models faking their retrieval scores

Oct 1

Forget Memorization: FutureBench Tests If AI Can Actually Predict Tomorrow

Jul 17

A 3-line code fix just tripled scores for DeepSeek and doubled Qwen on the math leaderboard

Feb 14

Arabic AI Benchmarks Fractured—A New Leaderboard Tries to Unite 700+ Models

Feb 10

GPT-4o's Reasoning Plummets 26 Points When Listening Instead of Reading

Dec 20

String-matching metrics are failing VQA models—even when answers are right

Jul 25

380M Arabic speakers get a leaderboard built to catch what English benchmarks miss

May 14

MMLU scores swing 28 points just by changing prompt format, study finds

Apr 30

New leaderboard ranks LLMs by how much chain-of-thought actually helps

Apr 23

UCLA's ConTextual benchmark: 506 text-rich tasks where GPT-4V beats humans once

Mar 5

Upstage's Korean LLM leaderboard hits 1,000 models in five months

Feb 20

New LLM leaderboard tests models on finance, legal work, and customer support

Jan 31