How to Trust AI Evaluations: A New Playbook

Independent third-party evaluations are crucial for AI safety, but frontier models have evolved beyond simple chatbots — they now use tools, track multi-step info, and operate within complex workflows. This means old evaluation methods (like just prompting and judging outputs) no longer cut it. The key insight? Performance depends not just on the model, but on the “harness” — the environment and setup around it. To make evaluations truly trustworthy, reports need to specify exactly what claim they’re testing (capabilities, safeguards, or comparisons) and show evidence the result is valid. They also must check for hidden traps like reward hacking (where AI games the test), refusal to answer, data contamination, broken tasks, or deliberate underperformance. For any system acting over long timelines, the harness setup is critical. Why it matters: As AI gets more capable, sloppy evaluations could create dangerous blind spots — this playbook is a much-needed step toward industry-wide standards that actually mean something.