Tagged: benchmarking

22 articles

Latest

BERT Inference on CPUs: AWS c5.metal Benchmarks Reveal Ugly Scaling

Needle benchmark shows search APIs hit just 55.7% of achievable quality on real agent queries

Why your MNIST model fails on real handwriting — and how adversarial data fixes it

MTEB puts 2,000+ embedding results across 56 datasets on one leaderboard

1,200 Rogue OpenAI Agents Formed a Collective to Cheat a Scorer That Never Existed

Liquid AI's Pipette Exposes Why Two 350M Models Keep 78% vs 34% Speed on Phones

Same MMLU benchmark, three wildly different scores for LLaMA

The Hidden Flaws in Object Detection Benchmarks: What Your AP Score Isn't Telling You

China's AI models now top global benchmarks, leaving US labs with a shrinking frontier

Stop benchmarking LLMs wrong: why time.perf_counter() alone is a trap

MoonshotAI’s PerceptionBench Exposes Where GPT-4o-Mini Still Can’t See Straight

ByteDance's EdgeBench reveals GPT-5.5's 22% edge, but time budgets flip the leaderboard

NVIDIA quietly ships srtctl: a YAML-to-SLURM compiler that makes LLM benchmarking reproducible

Not all correct AI answers are equal: Why Hugging Face is benchmarking the path, not just the result

Hugging Face locks down new speech datasets to stop AI benchmark gaming

NVIDIA dares the industry: reproduce our Nemotron Nano 3 scores yourself

BAAI's FlagEval Debate forces LLMs into adversarial showdowns, and the best models often lose

BigCodeBench stress-tests LLMs with 1,140 tasks and 99% branch coverage

Hugging Face's TGI Benchmark Tool Exposes the Latency-Throughput Tradeoff

GPT-3 Told a Pregnant Patient to Take a Drug That Could Harm Her Fetus

LiveCodeBench Exposes AI Coding Blind Spots With Time-Aware Contamination Tests

Upstage's Korean LLM leaderboard hits 1,000 models in five months