agent experience
You can't generate evals on autopilot
Writing eval criteria is hard, so you ask an LLM to write them. Minutes later, you have hundreds of reasonable-looking rubrics. But whose idea of a correct answer are they measuring?
Run Vally evals on GitHub Copilot in VS Code
Your skill worked when you tried it in GitHub Copilot Chat. After you change it next week, how will you know it still does?
Evaluating GitHub Copilot in VS Code just got easier
Your users run GitHub Copilot in VS Code, but most agent evaluation frameworks work with CLI agents. Close that gap, and you suddenly need to automate an IDE that changes every week.
If you can't automate your workflow, neither can your users' agents
You want to test whether an agent can deploy an app to your platform. But before you can check the agent's work, your eval harness has to click through a portal and an interactive sign-in. Whose problem is that?
Be honest about what your evals prove
Your coding agent scored 100% on its eval, and every criterion passed. Then someone tried to build the code it generated.