Some agent skills don’t have a right answer to check against, which got me rethinking what I recently wrote about whether every agent skill needs an eval. I argued that the need for evals depends partly on how a skill is used. If you build a skill for yourself, you can judge its usefulness through experience. If you distribute it to thousands of developers, a bad skill affects far more people, so you should have evidence that it works. But I started wondering whether some skills are simply the wrong kind of thing to evaluate.

Consider a brainstorming skill. Its purpose is to help you explore ideas and challenge assumptions, surfacing alternatives you might otherwise overlook. Unlike a skill that generates a configuration file, it doesn’t have a single correct output. One conversation might produce three ideas and another ten. One might take five turns, another fifteen. You can’t reasonably expect every conversation to follow the same path or reach the same answer, so I assumed that skills like this were poor candidates for evals. The more I thought about it though, the less convinced I became.

Start with what the skill promises

It turns out, that I was confusing how unpredictable the process is with whether its outcome can be measured. You can’t predict exactly what a brainstorming skill will produce, but you can still define what a successful result looks like. Suppose our skill promises to help developers explore different approaches to technical problems. We could build an evaluation scenario around a slow API, where the user asks the agent to brainstorm ways to improve its performance. Rather than specifying what the agent should say, we define the minimum expectations for a useful brainstorming session:

Criterion Expected evidence
Explores alternatives Identifies meaningfully different approaches, rather than variations of the same optimization
Considers tradeoffs Explains relevant costs, risks, or limitations of the proposed approaches
Challenges assumptions Recognizes that the actual bottleneck hasn’t been established and recommends measuring it before choosing an optimization

These criteria don’t prescribe the conversation. The agent is free to explore other ideas or ask questions we hadn’t anticipated. Because the criteria describe only what it must accomplish at minimum, doing more than expected costs it nothing. That’s deliberate: after all, a brainstorming session that surfaces something we didn’t think of is doing exactly what the skill promises.

Of course, the actual rubric needs more precision than this table. What counts as a meaningfully different approach? Which tradeoffs are relevant to a slow API? What shows that the agent challenged an assumption rather than merely mentioning it? The rubric needs to answer these questions explicitly enough that an evaluator applies the criteria the same way every time. Otherwise, an LLM judge fills the gaps with its own interpretation, so the score reflects another model’s opinion instead of the skill’s effectiveness.

Five turns or fifteen

Defining the minimum expectations also takes care of the variability that made me doubt conversational skills. A brainstorming session might take five turns or fifteen, with different questions along the way and ideas phrased in different words. If both conversations explore meaningfully different alternatives, weigh the relevant tradeoffs, and question where the bottleneck actually is, both succeeded. Their paths don’t need to match.

The turns still count, though. If one conversation reaches the outcome in five turns while the other needs fifteen, you’re paying for ten extra turns that didn’t improve the result. With agents, every extra turn means more inference, which costs time and money. A skill that produces a good result isn’t necessarily an efficient one. So evaluate the quality of the outcome separately from the cost of reaching it.

Compare against the base agent

Even if the brainstorming skill meets every criterion, does it actually help? To find out, you’d run the same scenarios with and without the skill. Perhaps the skill consistently explores more relevant alternatives or identifies tradeoffs that the base agent frequently misses. Perhaps it gets comparable results in fewer turns. Improvement doesn’t have to mean a higher average score. If the skill reliably reaches the expected outcome while the base agent only sometimes gets there, that consistency alone is worth codifying in a skill. But what if the base agent achieves the same results at half the cost? Then the skill adds no value in that scenario, even though it satisfies every criterion in its rubric.

I’ve experienced this firsthand. A while ago, I built gitload, a small CLI for downloading specific files from GitHub, and gave agents a skill for using it. I was convinced it was useful. Then I ran my first eval. Of roughly ten advertised scenarios, only one showed a benefit over the base agent. In the others, agents could accomplish the same tasks more effectively without gitload, which called the value of the CLI itself into question for agent workflows. Without that evidence, I might have kept investing in something that agents largely didn’t need. That’s why I now consider comparing a skill against the base agent essential. A skill that merely works hasn’t earned its place until it shows what it adds over the agent on its own.

Suppose our brainstorming skill consistently meets the criteria and beats the base agent at comparable cost. Have we proven that it makes developers more creative? No. Have we proven that its suggestions lead to better software? Not necessarily. What we’ve shown is that, for the scenarios and criteria we evaluated, the skill improves the brainstorming outcomes we care about. That’s a real basis for a decision, as long as we’re precise about what the eval demonstrates and what it leaves untested. An eval score isn’t universal proof of effectiveness, but being unable to measure every possible benefit is no reason to dismiss evaluation altogether.

Every skill needs evidence

Brainstorming looked like the clearest example of a skill that’s too subjective and conversational to evaluate. Once I stopped thinking about what the conversation should look like and started thinking about what it should accomplish, the problem became much more manageable. Define what the skill promises and build representative scenarios. Specify the minimum expectations in an unambiguous rubric. Then compare the results against the base agent on quality, consistency, and cost. None of that requires the skill to produce identical outputs across runs.

I’m increasingly convinced that every skill should have evidence supporting its value, including the skills you build for yourself. That doesn’t mean every skill needs an elaborate public evaluation suite. But the impression that something helps is a weak foundation for deciding whether to keep using it, especially when models and agent harnesses keep improving. A skill that adds value today can become unnecessary tomorrow. Without evals, how would you know?

The harder it is to define what success looks like, the more important it is to clarify what the skill actually promises. So when a skill seems impossible to evaluate, work out what you’re trying to prove. That’s where its eval starts.