Meta has released CyberSecEval 2, an open-source benchmark suite designed to measure how vulnerable large language models are to cybersecurity abuse — and the results show an industry that’s improving on some fronts while remaining dangerously exposed on others.
The headline number: average compliance with requests to assist in cyberattacks has dropped from 52% to 28% since the first CyberSecEval benchmark shipped in December 2023. That’s a meaningful shift in under a year, and it suggests model developers are actually paying attention to offensive security risks rather than treating them as an afterthought. Meta also found that general-purpose models tend to refuse malicious requests more often than code-specialized ones — though that gap has narrowed, which the company frames as code-focused models catching up on safety.
Prompt injection, however, is a different story. Meta’s testing found that conditioning LLMs against these attacks remains “an unsolved problem,” and the report is blunt about what that means for developers: don’t assume your model will follow system prompts when an adversary is feeding it adversarial input. That’s a significant warning for anyone building LLM-powered applications that touch sensitive data or external tools, especially given how aggressively companies have been shipping AI agents into production environments over the past year.
The benchmark also tested code interpreter abuse, where models are manipulated into executing malicious code in sandboxed environments, and capture-the-flag style exploitation challenges covering SQL injections and buffer overflows. On that last front, Meta’s assessment is sobering: even models with strong general coding skills can’t reliably solve end-to-end exploit challenges. That cuts against the occasional hype suggesting LLMs are becoming autonomous hacking tools. They’re not — at least not yet. But the prompt injection findings suggest the more immediate risk isn’t a model turning malicious; it’s a model being tricked by someone who understands its failure modes better than the developer who deployed it.
