Innovation Matters
Essays on AI, developer tools, and thoughtful software.
Evaluating GitHub Copilot in VS Code just got easier
Your users run GitHub Copilot in VS Code, but most agent evaluation frameworks work with CLI agents. Close that gap, and you suddenly need to automate an IDE that changes every week.
Agents removed the friction that built our technical literacy
An agent fixes a runtime version and an authentication problem on its way to getting your app running, then reports "Done." You saved twenty minutes, and you lost whatever those twenty minutes used to teach you.
If you can't automate your workflow, neither can your users' agents
You want to test whether an agent can deploy an app to your platform. But before you can check the agent's work, your eval harness has to click through a portal and an interactive sign-in. Whose problem is that?
Be honest about what your evals prove
Your coding agent scored 100% on its eval, and every criterion passed. Then someone tried to build the code it generated.
Your eval judge doesn't know what you mean
Does the solution follow best practices? Your LLM judge answers without hesitation and explains its verdict convincingly. So which best practices did it check?