Every agent skill should come with a nutritional label. We already have descriptions that tell us what skills do. The label should show whether a skill actually makes the agent better and what that improvement costs.
If you tell me that your skill improves how an agent writes documentation or reviews code, my next question is simple: how do you know? It works really well for me isn’t evidence. Neither is a compelling demo or a GitHub repository full of stars. Showing me two outputs and letting me decide that one looks better doesn’t prove much either. Without measurement, we’re still in trust me, bro territory.
Show me the tradeoff
A nutritional label gives me enough information to decide whether I should use the skill. At its simplest, I want to see whether adding the skill improves or harms the outcome. I also want to know how many tokens it consumes and what completing the task costs. All of those numbers need a baseline, so imagine an eval produces these results:
| Configuration | Lift / drag | Tokens | Task cost |
|---|---|---|---|
| Base agent profile | n/a | 48K | $0.82 |
| + skill | +14% | 72K (+50%) | $1.30 (+59%) |
The numbers are illustrative, but the tradeoff is tangible. The skill improves the outcome by 14% while consuming 50% more tokens and increasing the task cost by 59%. Is that a good result? It depends. For one task, the improvement might easily justify the cost. At scale, the extra inference might make the skill too expensive. That decision belongs to the person using the skill, and you can’t make it when all you know is that the skill improves results.
Seeing one result isn’t measuring the system
Some improvements seem so obvious that measuring them feels unnecessary. Imagine a skill designed to make writing more concise. You give the agent a 600-word draft, it returns 400 words, and the result reads better. Why would you need an eval when you can see that it works?
But what exactly did you see? You saw one shorter document without seeing that the agent might have needed three times as many tokens to produce it. You also don’t know whether, across 50 documents, it regularly removes information you wanted to preserve. Inspection shows the outcome you were looking for while hiding the system around it.
We notice the visible improvement and easily miss the cost or failures outside that one example. The shorter document catches our attention because that’s what the skill promised. Its token use and the information it removed are much harder to spot. A skill can improve the output in front of us while making the agent worse overall.
An eval can cost more than the answer is worth
Does every skill really need an eval though? This is where the argument gets inconvenient. Suppose you make a small skill that bumps a version before a release. It reads the current version, updates a couple of files, then runs the relevant checks. Do you really need to build an eval suite, execute dozens of runs, and maintain all of it as models change? Probably not. The eval could cost more than the uncertainty it removes.
If you use the skill once and it costs three times as much as it could have, the difference hardly matters. Run it 50 times a day and the economics change. Put it in a workflow used by a whole team and small inefficiencies start to compound. Publish it for thousands of people and something irrelevant in your terminal can turn into considerable wasted inference. Exposure changes the economics of evidence. The more often a skill runs and the more people depend on it, the more valuable it becomes to know what it actually does.
Cooking for yourself is different
If I cook something in my own kitchen, I don’t need a health inspector. I know the kitchen and I can taste the food. If I make something terrible, I’m the one eating it. The expectations change when I start cooking for others. Agent skills have a similar boundary, even if the consequences are nowhere near comparable.
I know my own setup. I know which model and harness I use, together with the tools available to the agent. I can inspect the result in the same context in which I created the skill. The moment I distribute it, I lose that knowledge. Someone might combine it with five other extensions or apply it to tasks I never considered. The number of variables grows together with the blast radius, making evidence more valuable.
Evidence isn’t certification
None of this means we need eval police. If someone builds a useful personal skill and puts it on GitHub, great. They can say that they made it for themselves and it seems to work. Others are welcome to try it as long as they understand what they’re getting: someone else’s judgment.
Compare that with a skill that publishes its evals and reports lift or drag alongside token usage and task cost. Its eval might not resemble your work, and the measured improvement might not justify the cost in your environment. Still, you have evidence you can inspect instead of outsourcing your judgment to the author or a persuasive README. The nutritional label gives you information you can judge for yourself.
Lift depends on the agent profile
A lift score belongs to a particular eval run with a particular agent profile. Change the model and the result might change. Change the harness and it might change again. The available tools and other installed extensions also influence what the agent does. A useful label must describe the conditions under which the measurements were taken.
That context lets us interpret the result and decide whether it applies to our work. Without it, the percentage looks more universal than it is. For example, a label could show the following profile and results:
Agent profile
- Harness: GitHub Copilot
- Model: Model X
- OS: macOS
- Extensions: A, B
- Eval: 30 technical-writing tasks, 5 runs each
Results
| Configuration | Lift / drag | Tokens | Task cost |
|---|---|---|---|
| Base profile | n/a | 48K | $0.82 |
| + skill | +14% | 72K (+50%) | $1.30 (+59%) |
Run the same eval against another profile and you might get a different result. Showing the profile makes those differences useful because we can see which conditions changed. This skill works is too broad a claim. Under which conditions does it work, and at what cost? That’s something we can measure.
The eval is the evidence
Even the label itself needs evidence. A badge claiming 27% lift at 3% additional cost looks impressive, but perhaps the author tested ten carefully selected tasks that align perfectly with the skill. A score without an inspectable eval is just trust me, bro with a decimal point.
The label should be an entry point into the evidence behind it. Show me the tasks and the baseline. Explain how you scored them and which agent profile ran them. Include the number of runs, so I can decide whether the evidence has anything to do with my work.
Authors only need to show what they tested and what happened. Proving that a skill works with every model and workflow imaginable would be impossible. The boundaries of the evidence are part of the evidence because they tell us how far we can trust the result.
The cost of not knowing
So, does every agent skill need an eval? No. A small personal automation that you can easily inspect might not justify the effort. Building and maintaining its eval could cost more than simply using the skill.
That equation changes when you run the skill constantly or let a whole team depend on it. It changes again when you distribute the skill to people whose environments and use cases you don’t control. At some point, it seems to work isn’t enough because the cost of not knowing starts to matter.
Claims that a skill is amazing or that everyone should install it tell me very little. Show me what happened when you measured it and what the improvement cost. Show me the eval, then let me decide whether it’s worth eating.