Why Most Evals Are Bad

Evals are an extremely important part of building, evaluating and improving AI models. A standard model training cycle works something like this:

1. Determine an important capability you'd like your model to have
2. Build an extremely high-quality eval that measures progress towards that capability
3. Try a lot of different training interventions and keep what moves the needle on the evals

When training the model, you are going to spend a lot of compute and time experimenting with different approaches to improve on the eval. It would be very upsetting to learn that the evals you were optimizing on were broken after you had invested a lot of resources into it.

Yet most evals being built are bad. FrontierBench, the next iteration of Terminal-Bench, recently wrapped up its call for tasks for the initial release. The project received over 500 task submissions but over 400 of those submissions were rejected and only 74 have been accepted for the first version of the benchmark. I was one of the task reviewers on this project and saw a few patterns first-hand that led to rejections.

What makes an eval bad?

There are many ways in which evals can stop providing signal on capabilities and become bad - these are some of the most common themes I have seen:

Weak verification

Models can fail at a task due to capability gaps but still be rewarded because of a verifier that's not configured correctly. Designing a verifier that gives the reward if and only if the intended capability is leveraged is much harder than it sounds.

For example, the configure-git-webserver task in Terminal-Bench 2.1 provides this instruction:

Configure a git server so that I can run on my computer
    git clone user@server:/git/server
    echo "hello world" > hello.html
    git add hello.html
    git commit -m "add hello.html"
    git push origin master
And have this data then be pushed to a webserver running on port 8080 so if I run
    curl http://server:8080/hello.html
then I see the output "hello world".
I'll setup login with the server to work, you don't have to worry about that.

In several trajectories we analyzed, agents failed to configure the git server at all but left behind a test server serving "hello world". The verifier, which effectively only checks that hello world is being served at the endpoint, rewards these agents anyways. A stronger verifier would write random content in hello.html on each run and then verify that the stated endpoint serves that exact random text, granting the reward if and only if the full pipeline worked end-to-end.

Misleading Spec

This is a common failure mode in adversarially generated evals where you're trying to build evals that models fail on instead of building evals that are representative of a valuable capability. In this failure mode, the prompt or the surrounding context in the environment (code, comments or docs) ask the model to do something that contradicts what the verifier expects.

In an adversarial eval development process, you would have people or agents that try different ideas and iterate on tasks until models fail on them and the task passes a fairness check. In this structure you end up building tasks that models fail on not because of true capability gaps but because of specific nuances about how current models interpret a specific ambiguous instruction. These tasks are also very hard to catch because they have a lot of complex domain-specific terminology that makes them look hard. AI review of the task might also fail here because these tasks could have been built with an AI reviewer in the loop so they're also adversarially optimized for AI reviews.

Reward hacking

Models are crafty: they'll mine git history for bugfixes, look up upstream patches online, overwrite your reward functions, and stub out pytest with a conftest.py that skips every test. If there's an easier path to the reward than the intended solution, they'll find it.

Without additional checks where an agent reads the trajectories to flag reward hacking, the eval can reward wrong capabilities that don't translate well to real world. Even with a check like this in place, you'd waste compute if most successful trajectories get flagged for reward hacking because the task makes it easier to reward hack than to develop the desired capability.

Tasks should be designed to minimize the hack surface from the start. An adversarial hacker that actively tries to find gaps in the eval development loop can help save a lot of time on debugging, task hardening and compute later on.

Incomplete Spec

In this case, the verifier tests for something that the prompt or the context never specifies. This could be an issue with the prompt or the verifier where it's falsely assumed that the test checks for a property that clearly follows from the spec.

This can also show up in benchmarks based on pull requests in public repos such as SWE-Bench Verified as the PR description might not fully capture the functionality being added or the author/maintainer may batch in a change that's not covered in the PR description.

Overstrict Verification

Verifiers should check objective outcomes instead of enforcing specific methodologies that lead to those outcomes. Many eval failures come from the latter, and it's not very bitter-lesson-pilled: given enough compute, models may discover better ways of solving the same problem. Computers play chess far better than humans, but in a style nothing like ours. An outcome-based verifier would reward that; one that checks for similarity to grandmaster moves would not. Once a good process monitoring pipeline is established that ensures models are leveraging the desired capability to solve the task, the model should be given free rein to explore.

There are several other ways in which an eval could be broken. Having agents review eval trial trajectories is gaining wider adoption and is showing some initial promise. Unfortunately, models still miss things here and can be deceptively confident after reading specific parts of the trajectory and coming to an incorrect conclusion. So humans still need to be in the loop to sample and review trajectories, to ensure the evals are evaluating what we aim to measure.

After building and reviewing a fair number of evals, I've found that good evals have the same shape usually: they are easy to understand, have a simple prompt (few lines at most) and are simple to verify, but require a lot of work. Rewriting git in zig or migrating an app from MySQL to Postgres without downtime are a couple of examples of such tasks.

Building good evals is really challenging. This is because frontier models have become really good and trivial tests don't cut it anymore. But most of the value that LLMs promise was always going to be in real-world tasks that are fairly complex and hard to model in an eval setting. There is a lot that needs to be built in evals and the infrastructure for it to enable us to model our most economically valuable tasks. This is one of the most significant risks to AI progress. Models can't get better at our most valuable problems until we get better at how we do evals.

If you're excited about researching gaps in model capabilities with a high degree of rigor, reach out to us at work@boolean.ai.