New Playbook for AI Evaluation: Harnesses and Context Define Capability Benchmarks

Article Summary

This detailed analysis provides a shared playbook for conducting independent, trustworthy third-party evaluations of frontier AI models. It argues that traditional chatbot-style evaluations are insufficient because modern models operate within complex 'harnesses'—environments that allow for tool use, state tracking, and multi-step workflows. Therefore, evaluation results must now explicitly detail the claim being tested, the specific setup (harness), and evidence addressing potential biases like reward hacking or data contamination. The recommendations emphasize that a standardized setup might understate a model's true capability, while controlled comparison requires fixing the environment to ensure fairness. Key recommendations include using tailored harnesses for strong capability elicitation and reporting resource dependency (e.g., cost per solve) rather than just fixed success rates.

Key Points

• Evaluation reports must explicitly detail the claim being tested and the specific 'harness' (the surrounding setup) used to validate the result.
• Model performance is critically dependent on the harness, meaning that standardized tests can significantly underreport a model's true capabilities if the environment lacks necessary tools or context preservation.
• Evaluators must adjust their reporting to reflect resource dependency, such as reporting performance per unit of compute or effort, rather than assuming a fixed capability ceiling.
• The playbook identifies common pitfalls—such as reward hacking, contamination, and sandbagging—that must be accounted for to ensure evaluation results are valid and trustworthy.