If you build skills or MCP servers for developers who use the Claude Code CLI, you’ve probably tested them by running claude in your terminal and checking whether it did what you expected. A manual check like that covers only the moment you ran it. It can’t show you whether your skill still works after you change it, or whether Claude Code would’ve solved the task just as well without it.

Yesterday, I released vally-vscode for evaluating GitHub Copilot in VS Code. Today, I’ve released its sibling: vally-claudecodecli, an executor plugin for Vally that runs your evals against the Claude Code CLI. It runs the real claude executable, headless, in a workspace that Vally prepares for each trial. You describe the scenario and how to grade it, and Vally takes care of the rest.

Why build on Vally

Vally is an eval platform from Microsoft. Every run in Vally moves through the same pipeline: a stimulus goes in, an executor runs the agent, the trajectory records what happened, and graders score the outcome. Executors are the swappable part of that pipeline, so adding the Claude Code CLI as an executor lets you reuse the rest of Vally as is. You get prepared trial workspaces and built-in graders, and after a run, you explore the results in Vally’s reports.

Because the executor determines which agent you evaluate, your scenarios and graders stay the same when you switch agents. That helps when some of your users work with the Claude Code CLI and others with GitHub Copilot, because the same model can behave differently depending on the agent it runs in. Run one eval against both, and when the results differ, you know that the difference comes from the agent rather than from a change to the eval.

Run your first eval

You need Node.js 22.15 or newer and the Claude Code CLI on your PATH, which you can check by running claude --version. Sign in by running claude once and completing /login, or set the ANTHROPIC_API_KEY environment variable. Then, add the Vally CLI and the executor to the project that contains your evals:

npm install --save-dev @microsoft/vally-cli vally-claudecodecli

Next, describe what you want to evaluate. A Vally eval is a YAML file that lists stimuli, which are the prompts you send to the agent, together with the graders that score each outcome. The following eval, saved as eval.yaml, asks Claude Code to create a file and then checks the workspace for the result:

name: claude-file-creation
defaults:
  executor: claude-code-cli
  model: sonnet
  timeout: 5m
stimuli:
  - name: creates-the-requested-file
    prompt: "Create hello.txt containing exactly: Hello from Vally"
    graders:
      - type: file-contains
        config:
          path: hello.txt
          value: Hello from Vally

The model property accepts anything that claude --model does: an alias such as sonnet or opus, or a full model name. Notice that the grader looks at the file on disk rather than at Claude’s response. Agents sometimes report that they completed a task when they didn’t, which is why your graders should check what actually happened.

Start the eval with the following command. The --executor-plugin option tells Vally which plugin provides the claude-code-cli executor that your eval uses. If you don’t want to pass it on every run, list the plugin under plugins.executor in your project’s .vally.yaml file instead.

npx vally eval -e eval.yaml --executor-plugin vally-claudecodecli

Vally runs each trial in a fresh workspace and grades the result. Its reports also show what Claude Code did along the way, including its tool calls and the tokens it used, so you can see how it got there and what it cost. To cap what each trial can spend, set the maxBudgetUsd option in the executor’s configuration.

Terminal showing an eval.yaml that uses the claude-code-cli executor, followed by a vally eval run that reports PASS for claude-file-creation

Unattended evals can’t wait for someone to approve tool calls, so by default, the executor runs Claude Code with bypassPermissions. That means that Claude Code runs commands and edits files without asking, with your account and credentials, so only run evals that you trust, including their fixtures and MCP servers. If you want to evaluate a more restrictive setup, set the permissionMode option in the executor’s configuration. Also, each trial’s artifacts keep Claude Code’s raw output, which contains your task and everything the agent generated, including local paths, so review them before you share them with anyone.

From here, add the skills and MCP servers that you built to the eval, and write graders that check whether Claude Code got the task right. For code, that can be your own tests, which Vally keeps away from the agent until it finishes, so Claude Code can’t write its code to fit them. Vally’s documentation covers how to set this up. As for what to grade, describe the user’s success rather than your product’s behavior.

Know what you’re measuring

The executor runs the Claude Code CLI the same way you do in your terminal, and it doesn’t isolate it from your machine. Each trial loads your user, project, and local settings, which bring along your CLAUDE.md files, your installed skills, and whatever else you’ve configured. That’s what keeps your sign-in working, and it’s realistic, because your users run Claude Code with their own setup too. But it also means that your configuration is part of the environment you measure. Instructions that you once added to your user-level CLAUDE.md can be the reason why an eval passes. MCP servers are the exception, because Claude Code loads only the ones that you list in the eval.

So keep your configuration consistent when you compare runs, or turn off customizations like your hooks and CLAUDE.md files by adding extraArgs: ["--bare"] to the executor’s configuration. In bare mode, Claude Code ignores the sign-in from /login, so set ANTHROPIC_API_KEY instead, and see the readme for other ways to narrow what Claude Code loads. With a clean baseline in place, run the same eval with and without your skill or MCP server, and see whether it’s worth the tokens it adds.

If your users work with the Claude Code CLI, that’s where you should evaluate what you build for them. vally-claudecodecli is available on npm, with the full list of supported Vally features and configuration options in its readme. It’s open source, and I’d love to hear how you use it, so share your feedback and ideas on GitHub.