Tagged: benchmarks

Berkeley's CUA-Lite kills the VM tax: OSWorld runs in Docker, 4.6x more desktops

GPT-6 Astra tops Epoch AI but stalls at Artificial Analysis — ARC-AGI-3 sees a 62.7% breakout

OpenAI Ships GPT-6 Astra, Claims It's the First Model in the AGI Era

TensorFlow's XLA compilation makes Hugging Face text generation up to 100x faster

Google's EnvHarness bends frozen benchmarks to each AI's flaws, boosting scores 9 points

Open ASR Leaderboard adds Hindi and Indian English — 4,888 speakers, 12 attributes each

Cohere's Parse Scores 79.2 on a Benchmark That Skips the Hard Parts

AI Won't Replace Junior Devs: The 4 Conditions Failing Hard

AWS Inferentia2 cuts Hugging Face model latency 4.5x over NVIDIA A10G GPUs

Nvidia's 3,400 token/sec Groq 3 LPX record needs 64 chips to beat Cerebras's two

GPT-4's 'human-like' labels fail to match real human raters in new study

AI Safety Scores Are Gamed by Overblocking—and Tests Can't Catch It

Adding 'beliefs' to world models lifts AI action prediction by 24 points

Falcon 180B: The 180B-parameter open model beating GPT-3.5 on MMLU

AI Agent Search APIs Benchmarked: Parallel Cuts Token Use 40% vs Basic

AI Agents Had $3,000 and Six Days to Write a Top-Tier Research Paper. They Failed.

GPU Allocator Beats FIFO by 33 Points Utilization on Identical Hardware

No AI model can truly see: best score on new perception test is 59.7%

Google's Gemini 3.7 Flash arrives 3 weeks after its predecessor, slashing coding costs in half

Grok 4.6 ties GPT-5.6 Sol at 61 points while costing 60% less

Microsoft's MAI Code 1.1 Flash gets crushed by DeepSeek on benchmarks and cost

xAI's Imagine 2.0 scores 1,439 Elo, nipping at OpenAI's heels in image benchmarks

Qwen3.8 Max catches Claude Opus 4.8 but burns 15x the tokens to do it

Meta's new Muse Spark 1.2 undercuts rivals at 20 cents—if you hand over your data

Reflex's XY charts 100M points in 0.08s, shrinks HTML exports by 1,000x

Alibaba’s Qwen3.8-Max autonomously ran a business for a year and beat 87% of human coders

Alibaba's Qwen3.8-Max hits 86.6 on Terminal-Bench, open weights drop next week

Supabase open-sources an AI agent benchmark — and 78% of coding agents flub auth without a skill file

GPT-5.6 Sol hits 38.3% on ARC-AGI-3, but only after OpenAI tweaks the rules

OpenAI’s agents cut scientific code runtimes 60x, but the real bottleneck is verification

Kimi K3 delivers 2.5x compute efficiency but flunks UK cyber test

Kuaishou's KAT-Coder cracks 100K repo tasks by fixing a 16% infrastructure error

Claude Opus 5 quadruples ARC-AGI-3 reasoning record, hitting 30.2%

Sakana’s Fugu-Cyber nudges past GPT-5.5 on one security benchmark—but it’s still gated

Marker 2 converts PDFs 5× faster than MinerU while scoring higher on AI2's benchmark

Claude Opus 5 ships with thinking on by default, breaks old API calls

Sakana's Fugu router now beats a model it can't even access, claiming 7.9-point gain

Unsloth crushes MoE training with 7.3x speedup, Axolotl hits 1.45x on Qwen3.5

Poolside's Laguna S 2.1 crushes DeepSeek on DeepSWE with one-sixth the active parameters

Alibaba's Qwen TTS edges past rivals in quality, but crawls at 16 chars/second

OpenAI reveals 3 moonshot projects to measure 'true' AI

Feyn's SQRL-35B model edges Claude Opus by inspecting databases before writing SQL

Why your next RL agent might need a safety net

Kimi K3 tops frontend coding but flunks advanced math

OpenAI's 16-Game Gauntlet Exposes RL's Memorization Problem

OpenAI's Codex solves 70% of coding puzzles by guessing 100 times

Bigger AI models lie more, new TruthfulQA benchmark finds

Fill-in-the-middle training doesn't hurt, it helps: new defaults for LLMs

OpenAI's o1 model beats PhDs and cracks top 500 in Math Olympiad

Murati's Thinking Machines drops Inkling: US best, but trails China

OpenAI Finds SWE-bench Is Broken—So They're Fixing It

OpenAI drops o1-preview: A PhD-level reasoner that thinks first

OpenAI's SimpleQA Exposes GPT-4o's 40% Factuality Score

OpenAI's CUA sets benchmark records but still fumbles 62% of computer tasks

OpenAI's SWE-Lancer tests if AI can actually earn $1M on Upwork

OpenAI's New Pioneers Program Aims to Fix AI's Benchmark Obsession

OpenAI's o3 and o4-mini can now 'think' with images, not just see them

OpenAI drops o3 and o4-mini: Reasoning models that actually use tools

OpenAI Says the Way We Score AI Actually Rewards Lying

OpenAI admits its own benchmarks are useless for 80% of the world

Claude Code claims 97% build success on Java migrations — but 24% of those apps were quietly broken

229,000 benchmark scores now link directly to Hugging Face model cards

ASR models fail hard in real rooms — Treble and Hugging Face just proved it

GLM-5.2 crushes open-source coding leaders, lands within 4 points of Claude Opus 4.8

A 0.6B model just beat SAM 3 on grounding—here's the simple trick that made it work

Meta and Hugging Face’s OpenEnv proves AI agents still can’t handle a simple calendar

Hugging Face kills black-box leaderboards with decentralized evals

H Company's Holo2 nails 78.5% on tough UI benchmark by looking twice

NVIDIA's Conformer-LLM hybrid tops ASR accuracy, but open-source still trails in long-form audio

14,000 coding battles reveal o3-mini rules the BigCodeArena leaderboard

Cohere's new RTEB benchmark catches models faking their retrieval scores

Meta's GAIA 2.0 benchmark humbles GPT-5 and Grok-4 with noisy, time-sensitive agent tasks

Ettin benchmarks prove encoder models crush decoders at 4X the efficiency

NeurIPS 2025 bets $16K that benchmarks, not loss curves, can judge baby LLMs

Hugging Face's ScreenSuite stress-tests 5 AI models on pure vision — no DOM cheating

H Company drops open-source Holo1 VLMs that slash web automation costs to $0.13 per task

DeepSeek drops V3-0324 with MIT license and a 19.8-point math leap

OlympicCoder 7B, a tiny open model, just beat GPT-4o and Claude 3.7 at live coding

Gemma 3's 27B model punches into top 10 of Chatbot Arena, beating Gemini 1.5 Pro

A 32B model just crushed Claude 3.7 Sonnet on Olympiad coding problems