Innovation Matters
Essays on AI, developer tools, and thoughtful software.
Would you check an agent skill's nutritional label before installing it?
An agent skill for SharePoint developers now ships with a label that shows what it does to seven different agent setups. Would you want to see one on every skill you install?
What if your eval rubric is the problem?
After we updated an agent skill, our evals showed the same success rate at nearly twice the cost. Reverting the change looked like the obvious decision. So why didn't we?
I let LLMs play a 1988 detective gamebook 646 times
In a hot-air balloon with a storm coming, almost every model made the sensible choice and landed. About ten decisions later, that choice cost them the case.
You can't generate evals on autopilot
Writing eval criteria is hard, so you ask an LLM to write them. Minutes later, you have hundreds of reasonable-looking rubrics. But whose idea of a correct answer are they measuring?
Run Vally evals on GitHub Copilot in VS Code
Your skill worked when you tried it in GitHub Copilot Chat. After you change it next week, how will you know it still does?