evals
I let LLMs play a 1988 detective gamebook 646 times
In a hot-air balloon with a storm coming, almost every model made the sensible choice and landed. About ten decisions later, that choice cost them the case.
You can't generate evals on autopilot
Writing eval criteria is hard, so you ask an LLM to write them. Minutes later, you have hundreds of reasonable-looking rubrics. But whose idea of a correct answer are they measuring?
Run Vally evals on GitHub Copilot in VS Code
Your skill worked when you tried it in GitHub Copilot Chat. After you change it next week, how will you know it still does?
Evaluating GitHub Copilot in VS Code just got easier
Your users run GitHub Copilot in VS Code, but most agent evaluation frameworks work with CLI agents. Close that gap, and you suddenly need to automate an IDE that changes every week.
Be honest about what your evals prove
Your coding agent scored 100% on its eval, and every criterion passed. Then someone tried to build the code it generated.