Innovation Matters
Essays on AI, developer tools, and thoughtful software.
You can't generate evals on autopilot
Writing eval criteria is hard, so you ask an LLM to write them. Minutes later, you have hundreds of reasonable-looking rubrics. But whose idea of a correct answer are they measuring?
Run Vally evals on Claude Code CLI
Your skill worked when you tried it in the Claude Code CLI. Would the CLI have gotten the task right without it, and for fewer tokens?
Run Vally evals on GitHub Copilot in VS Code
Your skill worked when you tried it in GitHub Copilot Chat. After you change it next week, how will you know it still does?
You're not falling behind because someone else ships more
Someone merged 100 PRs last week, and you're still working on the feature you started on Monday. You feel like you're not going to make it, but are you?
Evaluating GitHub Copilot in VS Code just got easier
Your users run GitHub Copilot in VS Code, but most agent evaluation frameworks work with CLI agents. Close that gap, and you suddenly need to automate an IDE that changes every week.