Turning a collection of books, papers, or documentation into an agent skill is increasingly easy. But once you have built the skill, how do you know it works?
Recently I’ve been working on turning a knowledge corpus into a skill. Because the corpus was substantial I decided to break it down into separate resources and use routing to guide the model towards the necessary pieces to answer the user’s question. I chose this setup to make it easy to distribute the skill and not depend on something like a search index.
No matter how you decide to store the knowledge in your skill, you’re left with the question: does it actually work? It’s tempting to assume that because the model turned the corpus into a skill, it closely followed your instructions to persist all facts and not lose any knowledge. But how do you know for sure? What’s the evidence?
You can read through SKILL.md and ask a few questions that you expect the source material to answer. When the responses look reasonable, you might conclude that the skill is ready. But those hand-picked questions tell you little about the material you didn’t test. Does the skill cover the corpus, and can it consistently guide the agent to answers with the required accuracy and detail?
Trust but verify
To verify that knowledge skills work as intended, I built skillfid (from skill fidelity): a CLI that lets you test whether a skill built from a source corpus helps an agent answer questions grounded in that material, and measures what the skill adds over the model alone.
skillfid focuses on knowledge skills: agent skills that make a body of knowledge available to an agent and guide it toward the right information. Your source might be product documentation or a collection of books and research papers. This is an important distinction: skillfid isn’t a general-purpose test framework for every kind of agent skill.
If you were to test your knowledge skill manually, there’s a chance you’d favor the material you remember and the questions you expect. You’d likely miss facts buried deep in the source or exceptions that change an otherwise correct answer. Connections spread across different sections are easy to overlook too. And the larger the corpus becomes, the harder it is to know whether your sample says anything meaningful about the whole skill.
Even a seemingly correct answer can omit a prerequisite or lose an important qualification. It might state the right idea at the wrong level of precision, or contain plausible claims that the source doesn’t support. skillfid evaluates answers against question-specific criteria, so you can inspect failures in correctness and completeness, including the precision and detail the question requires.
Measure what the skill adds
The point of skills is to bring models knowledge they don’t have, whether it’s knowledge of processes or facts. So merely evaluating the model’s response is not enough. The underlying model might already know enough to answer the questions without the skill, and repeating the same facts in the skill basically wastes tokens and overstuffs the context window.
To address this problem, skillfid evaluates two conditions, starting with how the agent answers without the skill. This closed-book baseline captures what the model can do on its own. It then asks the same corpus-grounded questions with the skill available, and the difference between the two scores is the skill’s measured lift.
I used skillfid while developing a real knowledge skill from an internal corpus at work. The evaluation dataset contained 291 corpus-derived questions. The first version of the skill scored just over 70% which proved there was plenty of room for improvement. In the latest run, the agent scored 12.96% closed book and 91.27% with the skill, an uplift of 78.31 percentage points.
How it works
First, skillfid turns your Markdown corpus into an evaluation dataset. It inventories the knowledge in the source, then uses that inventory to generate questions and grading criteria linked to source evidence. Before creating the dataset, it verifies that the questions are answerable from the corpus and that the accepted knowledge is covered.
In parallel it also calibrates each question using an answer generated with access to the complete source. That answer must receive a stable, perfect criterion-level judgment before the question passes the publication gate. This checks the generated evaluation against the source before you use it to assess your skill.
Next, skillfid creates a reusable closed-book baseline for the dataset. You can evaluate different revisions of your skill against the same questions and compatible baseline without rebuilding everything each time. To reuse that baseline, your evaluation settings and runtime versions must match.
Finally, you run the skill-assisted evaluation and generate a self-contained HTML report with a separate local command. The report shows the scores and uplift, then traces failures to individual grading criteria and source evidence. Where possible, it diagnoses what might have gone wrong and points to places in the skill that you could improve.
You can give the report to an agent and inspect the changes it proposes. After updating the skill, evaluate it again against the same dataset to see whether the changes helped. Repeat that loop against a quality threshold you chose, rather than editing until the skill feels better.
Try it on your knowledge skill
skillfid requires Node.js 24 or later and an authenticated GitHub Copilot CLI. Install skillfid from npm:
npm install --global skillfidIf you don’t have a knowledge skill readily available, take a look at the Canopy example in the repository. It’s a synthetic scenario built around a distributed build cache. It includes the corpus and generated dataset, plus the baseline and skill runs with an HTML report. You can inspect these artifacts without making a model call and compare the original skill with a revision informed by the evaluation findings.
This is the first preview version of skillfid, so expect rough edges. Also, keep in mind, that model-backed dataset generation and evaluation use inference, and costs can add up as the corpus and number of trials grow, especially with more powerful models. Start with the included example to understand the workflow, then choose your evaluation configuration deliberately.
If you maintain a knowledge skill today, try skillfid on it and see whether it adds what you think it does. If you don’t have one yet, explore Canopy and its report. Issues and feedback are welcome on GitHub.