Your coding agent just scored 100% on an eval, and every criterion passed. There’s only one problem: the code it generated doesn’t build. How is that possible? The eval checked 20 aspects of the implementation. The agent used the right APIs and followed the expected patterns. It created the required files and included the necessary configuration. Every one of those checks passed, but nobody tried building the application.
The eval did exactly what it was designed to do. It verified the criteria it was given, and the agent met all of them. The problem starts with what we infer from that result. A 100% eval score means that 100% of the evaluated criteria passed. It doesn’t tell you whether the agent completed the user’s task.
The temptation of proxies
Ideally, an eval verifies whether the agent achieved the user’s goal. If the user asks for a working application, you verify that the application works. In practice, that’s easier said than done. Consider evaluating an agent that builds and deploys a cloud application. To verify that it succeeded, you’d need to build the generated code and provision the infrastructure. Then you’d deploy the application and exercise its functionality before cleaning everything up. Along the way, you’d have to handle failures and unpredictable outputs, sometimes for applications whose implementation you can’t anticipate. A harness like that can take considerable time and money to build.
So we compromise. Instead of verifying that the application works, we check whether the generated code looks plausible. We inspect the files and configuration that the agent produced and compare its API usage against the patterns we expect. These checks become proxies for functional success, and there’s nothing inherently wrong with that. Engineering is full of tradeoffs, and evaluating agents is no exception. Sometimes direct verification is impractical, and a proxy is the best evidence you can reasonably get. The trap is forgetting that we’re measuring a proxy and communicating the result as proof of success.
Validation isn’t the same as success
Agent skills show the same pattern. You can automatically verify that a skill’s frontmatter is valid and that its structure follows the specification. You can even check whether the agent invokes the skill or whether its output contains certain strings. These checks are useful because they catch structural problems and regressions, much like linting or static analysis does in traditional software development. But none of them prove that the skill helps the agent complete the user’s task.
Nobody claims that an application works because it passes linting. We treat static analysis and builds as different evidence from functional tests, and we validate performance and deployments separately again, because each of them tells us something different about the software. Agent evaluations deserve the same discipline. A skill that passes structural validation meets certain technical requirements. An agent invoking the skill is evidence that the skill was invoked. That’s it. Neither, on its own, proves that the user got what they asked for, and even a comprehensive collection of proxy checks might miss the one thing that matters most.
What does your eval actually prove?
Before you publish an eval result, complete this sentence, and be precise about it:
When this eval passes, we know that…
If your eval checks implementation patterns without building the code, an honest answer sounds like this:
When this eval passes, we know that the agent generated code that satisfies our implementation criteria. We haven’t verified that it builds or runs.
That’s a defensible claim. What you shouldn’t say is that the agent built a working application, because your eval never checked that. The same goes for the cloud application from earlier. An agent can generate infrastructure definitions and deployment scripts that look entirely correct, but until you deploy and validate the application, you don’t know whether it works in the target environment. If a working deployment was what the user asked for, that’s a significant gap in your evidence. An eval doesn’t have to verify everything. You might deliberately accept that gap because direct verification costs too much, and that’s your tradeoff to make. But the confidence you communicate should never exceed the evidence you’ve collected.
Don’t let the nuance disappear
Imagine a team that reports consistently high eval scores. The results look excellent. Management sees progress that justifies the investment, and everyone believes that the agent experience is improving. Meanwhile, users complain that the generated applications don’t work. How can both be true? Perhaps the eval scenarios don’t represent what users actually encounter, or the criteria verify implementation details while missing functional failures. Perhaps both. Either way, the team has been optimizing against its eval results while gradually losing touch with what its users experience.
That’s why a misleading eval can be worse than no eval at all. Without evidence, you know that you’re uncertain. With a reassuring score that nobody fully understands, you might stop asking whether things actually work. So when users report failures despite high eval scores, treat that as a reason to revisit both your scenarios and your criteria. Your eval might cover different situations from the ones your users face, or measure outcomes other than the ones they care about.
Preventing this is primarily the job of the people who design the evals. They know what they measured and which compromises they made, so they also know what remains unverified. You can’t expect everyone who reads the results to inspect the eval harness and reconstruct those limitations themselves. You can report them in a matrix of separate scores or in a short explanation. Either works, as long as the scope and limitations of the evidence travel with the results.
Be deliberate about what you measure
Proxies are often a practical response to how complex it is to evaluate real-world agent behavior. They’re still compromises, though, and you should make them deliberately. Whenever possible, evaluate whether the user actually succeeded. When that’s impractical, use the best evidence you can get and be clear about its limitations. Don’t confuse validation checks with outcome evaluations, and don’t let a perfect score imply more than you’ve verified. Before you celebrate your next 100% eval score, ask yourself: when this eval passes, what do we actually know? Then make sure that’s also what everyone reading the results understands.