AI Struggles with Real Knowledge Work, Fails 97% of Tasks

A new benchmark, the AA-Briefcase test, evaluates AI models on complex knowledge work projects involving thousands of fragmented source files. Despite top performers like Claude Fable 5, AI models consistently fail to fully solve tasks, with only 3% success rate. This highlights the limitations of current AI technology and underscores the need for more realistic evaluations. Why it matters: This study’s findings underscore the need for more practical and nuanced assessments of AI capabilities, as the current hype surrounding AI may not accurately reflect its true potential.