We have continuous integration and continuous deployment. AI coding agent extensions need continuous evaluation. If you build a skill, MCP server, or another extension for a coding agent, you need to know whether it keeps adding value as everything around it changes.
You can start by giving an agent a set of tasks without your extension, then repeat them with your extension available and measure the difference. Does your extension improve the outcome? That’s lift. Does it make the outcome worse? That’s drag. How many tokens did the agent use, and how much did the task cost? Run the evaluation often enough and you have evidence that your extension adds value. Then you ship it. There is however on problem: your evidence is out of date the moment you ship the extension.
Rolling stone gathers no moss
Suppose you evaluated your extension with a particular coding agent and model and found that it consistently improved the agent’s performance. Days later, the coding agent ships an update. Your extension hasn’t changed, so should you run the evaluation again? Of course!
The update might have changed the agent’s system prompt, context management, or tool selection. Any of those implementation details can affect how the agent discovers and uses your extension. Your code is the same, but the system you measured is different.
Then a new model becomes available. The model might follow your instructions better, or it might already know enough about the problem that your extension adds less value. It might use your extension more efficiently or consume far more tokens while doing so. You need to evaluate again.
We already work this way with continuous integration. We don’t run CI only when we change application code. Updating a dependency is enough reason to run the tests because the behavior of the system we’re shipping might have changed. AI coding agent extensions simply have more things outside their repositories that can change their behavior.
For an evaluation, I think of those things as an agent profile:
harness + model + reasoning level + operating system + extensions
I evaluate one or more tasks, with explicit evaluation criteria, against a set of these profiles. For every evaluation, I measure the same things: drag or lift, token use, and task cost. I also retain the agent’s trajectory. The score tells me what happened, while the trajectory helps me understand why, spot unexpected behavior, and form hypotheses for improving the extension. When something relevant in a profile changes, I evaluate again. That’s continuous evaluation.
Someone else’s extension can change yours
There’s one part of the agent profile that’s easy to overlook: extensions. Your extension is unlikely to be the only capability available to an agent. A developer might have your skill alongside several other skills and MCP servers. You didn’t choose those extensions, you don’t control them, and their authors don’t coordinate their releases with you. They’re peers.
One of those peers changing can change the measured value of your extension. Every token added to the context can affect an LLM’s behavior. Extensions can participate in discovery before either of them is invoked. If two extensions cover nearby territory, the agent might choose one instead of the other or invoke both. Their instructions might complement each other, conflict, or lead the agent down a completely different path.
A peer might even become good enough at solving a problem that the agent stops invoking your extension altogether. Your code and instructions haven’t changed. Your extension might still work exactly as designed when explicitly invoked, and yet its lift could have disappeared. Someone else can ship a change and change the value of your software. Continuous evaluation therefore needs to watch more than your own repository or the companies behind the model and coding agent. You also need to know what your extension is likely to encounter.
Continuous evaluation needs continuous discovery
You cannot evaluate every possible configuration. Even without extensions, the evaluation space is effectively infinite. Change the wording of a task and you’ve created another evaluation. Add models, reasoning levels, operating systems, coding agents, extension versions, and combinations of extensions, and the number of possible evaluations explodes.
The goal is to generate as much data as you can use and turn into meaningful results. That means deciding which agent profiles matter. And that requires continuous discovery.
Talk to your users. Just as you research the problems they’re trying to solve, learn about the environments in which they’re solving them. Which coding agents and models do they use? Which other skills and MCP servers do they commonly have available?
Pay attention to the ecosystem too. Which extensions are becoming popular or are likely to be installed alongside yours? Which ones occupy adjacent parts of the workflow? When you discover a peer that your extension is reasonably likely to encounter, add that composition to the profiles you evaluate.
Perhaps nothing happens, or the two work beautifully together and your lift increases. Your extension might be selected less frequently. The combination might even produce worse results. You don’t know until you evaluate it, and six months later you might need to evaluate it again.
Discovery informs the evaluation matrix. Evaluations produce scores and trajectories, which reveal odd behavior and suggest hypotheses. You improve the extension while its peers and the rest of the agent profile keep changing. Discovery tells you which configurations matter next. The eval suite is a living representation of the environment in which you believe your extension operates.
Evaluation needs to measure value
Traditional regression testing mostly asks whether the software still works after something changed. For an AI coding agent extension, that question doesn’t go far enough. The extension can still work perfectly and no longer be useful.
A newer model might perform the task just as well without it. A peer extension might provide the same information more effectively. Your extension might still improve quality, but increase token use so much that the improvement isn’t worth its cost. So the right question to ask instead is: does the extension still add value?
That’s why I care about drag and lift alongside token use and task cost. Together they tell me whether making the extension available improves the agent enough to justify what it consumes. Unlike whether a function returns the expected value, that answer isn’t a permanent property of the extension. It’s a measurement made under particular conditions, and when those conditions change, the measurement can change too.
Your evidence has a shelf life
None of this means every skill published on GitHub needs an elaborate continuous evaluation infrastructure. You can build a useful skill, publish it, and tell people to try it. But if you want to claim that your extension makes coding agents better or more efficient, you’re making an empirical claim. You need evidence.
Because the system around your extension keeps changing, evidence gathered once doesn’t establish that claim forever. You can choose not to evaluate or reevaluate when a model changes. You can ignore what other extensions your users have installed. There’s nothing inherently wrong with that, but you’re operating in trust me, bro territory.
Continuous evaluation moves the conversation from I think this helps to here is evidence that it helps under the conditions my users actually encounter. Continuous discovery keeps those conditions grounded in reality. Continuous integration acknowledges that software keeps changing. Continuous evaluation acknowledges something slightly stranger: your AI coding agent extension doesn’t have to change for its value to change.