Innovation Matters
Essays on AI, developer tools, and thoughtful software.
Be honest about what your evals prove
Your coding agent scored 100% on its eval, and every criterion passed. Then someone tried to build the code it generated.
Your eval judge doesn't know what you mean
Does the solution follow best practices? Your LLM judge answers without hesitation and explains its verdict convincingly. So which best practices did it check?
As models get better, your agent tools should get smaller
You wrote your agent skill to fix what the agent kept getting wrong. What happens to those instructions when a new model stops getting it wrong?
Your eval should describe the user's success, not your product's behavior
Your eval checks whether the agent called the tool you built, and every run passes. What does that tell you about the developer who asked for help?
Stop sensationalizing
Somewhere along the way, a settings page became an experience and a fix became an unlock. We use big words for small things when there's no need to.