When I started building evaluations for GitHub Copilot in VS Code, I quickly ran into a problem: how do you automate an IDE? Knowing what to evaluate is only part of the challenge. You also need to drive VS Code reliably enough to run those evaluations, and keep them running as VS Code changes.
Evaluate the agent your users work with
If you want to understand how well an AI coding agent performs for your users, you need to evaluate it in the environment they use. GitHub Copilot in VS Code isn’t the same agent as GitHub Copilot CLI or the Copilot app. Each of them runs in a different environment with its own system prompt and set of tools, and those differences can affect the outcome. So you can’t assume that results from one represent the others.
The same applies when you decide whether an agent extension is worth keeping. Every extension you introduce comes with an inference cost. Some improve outcomes enough to justify that cost, while others add tokens and complexity without a meaningful improvement. The only way to tell them apart is to measure both the outcomes and the cost of representative scenarios in the environment your users work in. If that environment is VS Code, results from a CLI agent won’t be helpful unfortunately.
Automating VS Code is harder than it sounds
Most agent evaluation frameworks work with CLI-based agents. That makes sense, because you can start a CLI agent with a prompt and collect its output with little effort. But VS Code is different. It’s an IDE, and while it exposes commands for its functionality, controlling it from an external process is far from straightforward.
Initially, I used Playwright to automate VS Code. It worked, but it came with a maintenance cost. VS Code used to ship a new stable version roughly every month, and now it ships one roughly every week. Updates frequently required changes to our Playwright automation, so we were spending time fixing the evaluation infrastructure instead of evaluating agents. The underlying problem was that we were automating the UI. VS Code’s DOM isn’t a public automation API, and nothing guarantees that it stays the same between releases.
Introducing VS Code Controller
To solve this, I built VS Code Controller, an open-source extension that lets external processes control VS Code and GitHub Copilot Chat programmatically. Your code talks to the extension through a local JSON-RPC socket, and the extension executes VS Code commands on your behalf. That gives you a more stable interface, one that doesn’t depend on the IDE’s DOM structure. So instead of updating UI automation after every VS Code release, you can spend that time understanding how the agent behaves in representative scenarios and measuring whether each improvement justifies its inference cost.
VS Code Controller gives you the instrumentation layer for your evaluations. It doesn’t define scenarios, and it doesn’t judge or score agent outcomes. How you use it, and what you build around it, is entirely up to you.
If you or your audience use GitHub Copilot in VS Code, don’t rely exclusively on results from CLI-based agents. Extend your evaluations to VS Code and measure the outcomes there.
We’re currently updating VS Code Controller, and it will be available again shortly.