Recently, I introduced VS Code Controller, an extension that lets you control GitHub Copilot Chat in VS Code from an external process. It solved the hardest part of evaluating Copilot in VS Code: driving the IDE without brittle UI automation. But it deliberately stopped there: it doesn’t prepare workspaces or score outcomes, so if you wanted to run evals, you had to build everything around it yourself.

I’ve just released vally-vscode, which fills that gap. It’s an executor plugin for Vally that runs your evals against GitHub Copilot Chat in the VS Code desktop app. You describe the scenario and how to grade it, and Vally takes care of the rest.

Why build on Vally

Vally is an eval platform from Microsoft. Every run in Vally moves through the same pipeline: a stimulus goes in, an executor runs the agent, the trajectory records what happened, and graders score the outcome. Executors are the swappable part of that pipeline, so adding VS Code as an executor lets you reuse the rest of Vally as is. You get prepared trial workspaces and built-in graders, and after a run, you explore the results in Vally’s reports.

And because the executor determines which agent you evaluate, VS Code is no longer a special case. Your scenarios and graders stay the same whether you run them against Copilot in VS Code or another agent that Vally supports. Since the same model can behave differently depending on the agent it runs in, keeping the scenario constant while changing the agent is how you find out whether your results transfer to the environment your users work in. When a skill passes with one agent and fails with another, you know that the difference comes from the agent rather than from a change to the eval.

Run your first eval

You need Node.js 22.14 or newer and VS Code 1.140 or newer with GitHub Copilot Chat installed and signed in. The executor supports only stable VS Code, not Insiders, so that’s where you install VS Code Controller, which vally-vscode uses to drive Copilot Chat. Then, add the Vally CLI and the executor to the project that contains your evals:

code --install-extension waldek.vscode-controller
npm install --save-dev @microsoft/vally-cli vally-vscode

Next, describe what you want to evaluate. A Vally eval is a YAML file that lists stimuli, which are the prompts you send to the agent, together with the graders that score each outcome. The following eval, saved as eval.yaml, asks Copilot to create a file and then checks the workspace for the result:

name: copilot-file-creation
defaults:
  executor:
    name: vscode
  model: auto
  timeout: 5m
stimuli:
  - name: creates-the-requested-file
    prompt: "Create hello.txt containing exactly: Hello from Vally"
    graders:
      - type: file-contains
        config:
          path: hello.txt
          value: Hello from Vally

Notice that the grader looks at the file on disk rather than at Copilot’s response. Agents sometimes report that they completed a task when they didn’t, which is why your graders should check what actually happened. Checking the workspace also gives you a result that you can trust when you rerun the eval after changing your skill.

From here, you can add your skills, instructions, fixtures, and MCP servers, and replace auto with a specific model. Use the model name exactly as it appears in Copilot’s model picker. The executor checks which model served the request and fails the trial instead of silently accepting a substitute, so when you compare models, you know which one you measured.

Before you run the eval, configure Copilot to work without you. By default, Copilot asks for approval before it runs tools, and when an eval runs unattended, nobody is there to answer, so the trial stalls until it times out. To prevent that, add the following to your VS Code user settings:

{
  "chat.defaultToCopilotHarness": true,
  "chat.defaultConfiguration": {
    "mode": "autopilot",
    "approvals": "allowAll"
  }
}

These settings let Copilot run terminal commands and edit files without confirmation, with your credentials and permissions, so only run evals that you trust, including their fixtures and MCP servers. Because these are user settings, they also apply to your regular chats, so restore your preferences when you’re done evaluating.

With everything in place, close any other stable VS Code windows and start the eval with the following command. The --executor-plugin option tells Vally which plugin provides the vscode executor that your eval uses. If you don’t want to pass it on every run, list the plugin in your project’s .vally.yaml file instead.

npx vally eval -e eval.yaml --executor-plugin vally-vscode

For each trial, the executor opens VS Code in the workspace that Vally prepared, sends the prompt to Copilot Chat, waits for it to finish, and closes VS Code. If your stimulus has multiple turns, the executor sends them to the same chat, just like a developer following up on a response. Vally then runs your graders and reports the results.

Terminal running vally eval with the vally-vscode executor plugin next to VS Code, where GitHub Copilot Chat in Autopilot works on the eval prompt in the trial workspace

See how the agent got there

A passing grade confirms that the agent completed the task, but to understand how it got there and what it cost, you need to see what happened during the run. Out of the box, the trajectory of each trial contains only the prompts and Copilot’s responses. To see more, install Dev Proxy and trust its root certificate. Dev Proxy stores the certificate as rootCert.pfx, but the executor needs it in PEM format, so export it and point the executor to the exported file with the DEV_PROXY_PEM environment variable. On macOS, run:

export DEV_PROXY_PEM="$HOME/Library/Application Support/dev-proxy/rootCert.pem"
openssl pkcs12 -in "$HOME/Library/Application Support/dev-proxy/rootCert.pfx" -nokeys -passin pass: \
  | openssl x509 -out "$DEV_PROXY_PEM"

On Linux, the commands are the same, except that Dev Proxy keeps its certificate in ~/.config/dev-proxy:

export DEV_PROXY_PEM="$HOME/.config/dev-proxy/rootCert.pem"
openssl pkcs12 -in "$HOME/.config/dev-proxy/rootCert.pfx" -nokeys -passin pass: \
  | openssl x509 -out "$DEV_PROXY_PEM"

On Windows, you don’t need OpenSSL. Instead, export the certificate in PowerShell:

$env:DEV_PROXY_PEM = "$env:LOCALAPPDATA\Programs\Dev Proxy\rootCert.pem"
$cert = [System.Security.Cryptography.X509Certificates.X509Certificate2]::new("$env:LOCALAPPDATA\Programs\Dev Proxy\rootCert.pfx", "")
"-----BEGIN CERTIFICATE-----`n" + [Convert]::ToBase64String($cert.RawData, "InsertLineBreaks") + "`n-----END CERTIFICATE-----" | Set-Content $env:DEV_PROXY_PEM

These commands set DEV_PROXY_PEM only for the current terminal session, so start the eval from the same terminal, or add the variable to your shell profile to keep it. If Dev Proxy later regenerates its certificate, trust the new one and export it again, because the executor needs the certificate that Dev Proxy currently uses.

When this variable is set, the executor starts its own Dev Proxy instance to record Copilot’s API traffic. It then converts the recording to an ATIF trajectory that shows the agent’s reasoning and tool calls along with the tokens that each request used. You’ll find the trajectory next to the recorded traffic in each trial’s artifacts. With that information, you can answer the questions that a score leaves open: Did the agent use the tool you built for it? How many requests did it take to get to the result? Did your new instructions make the agent more effective, or just more expensive? Recordings contain your task and everything the agent generated while working on it, so review them before you share them with anyone.

Know what you’re measuring

The executor runs the same VS Code that developers use, which is both its purpose and its main constraint. Instead of running headless, it opens and closes the desktop app with your signed-in profile. And because VS Code runs one process per profile, trials run one at a time, and you can’t use stable VS Code yourself while an eval is running.

Running with your profile also means that your installed extensions, settings, organization policies, and available models are part of the environment you measure. That’s realistic, because your users run Copilot with their own setup too, but it also means that you need to keep that environment consistent when you compare runs. If you’re evaluating your own VS Code extension, keep it disabled in your profile and use the executor’s extensions option to enable it only for the trial workspace. That way, you can run the same eval with and without it, and see whether it’s worth the tokens it adds.

If your users work with GitHub Copilot in VS Code, that’s where you should evaluate what you build for them. vally-vscode is available on npm, with detailed configuration and troubleshooting guidance in its readme. It’s open source, and I’d love to hear how you use it, so share your feedback and ideas on GitHub.