“Does the solution follow best practices?” It’s a simple yes-or-no question, and it sounds like a reasonable evaluation criterion, especially when you’re using a capable LLM as a judge. After all, modern models know a great deal about software development. Surely they can recognize good code when they see it?
But with that one question, you’ve delegated several decisions to the judge. Which best practices? Which of them are relevant to this solution? How strictly should the solution follow them, and do some matter more than others? What counts as sufficient evidence? The judge will happily fill in the blanks, and it might even give you a convincing explanation for its verdict. Yet nothing guarantees that it will make the same assumptions next time, or that another model will reach the same conclusion.
Missing knowledge is the more familiar version of this problem. If the judge lacks domain knowledge about your technology, a vague criterion lets it score incorrect code as correct, so your criteria need to spell out what correct usage looks like. But even a judge with perfect domain knowledge can’t tell which of several reasonable interpretations you had in mind, because you never wrote that choice down. Knowing more only gives the judge more plausible ways to fill in the blanks. A binary question isn’t necessarily an objective question, and no matter how capable the judge is, it can’t stand in for a well-defined rubric.
Beware of hidden assumptions
Consider another seemingly reasonable criterion:
Does the solution handle user input?
Take a red pen and underline two parts: handle and user input. What does handle mean? Accept the input? Validate it? Sanitize it? Store it? Show an error when something goes wrong? And which user input are we talking about: a customer email address, a search query, an uploaded file? Every unanswered question is an assumption we’re asking the judge to make, and the more of them we leave open, the more room there is for its verdict to vary.
So we improve the criterion:
Does the solution validate customer email addresses?
Better. We’ve narrowed the scope and clarified what we expect the solution to do, but we’re not finished. What makes an email address valid? Is checking for an @ symbol sufficient? What should happen when validation fails, and does the solution need to keep invalid data from being persisted? We’ve removed some ambiguity, but the judge still has to decide what successful validation looks like.
Describe success without prescribing the implementation
Let’s refine the criterion further:
The solution must reject customer email addresses that are empty, lack an
@symbol, or have no domain after the@symbol. Invalid addresses must not be persisted, and the user must receive an indication that the input was rejected.
This isn’t a comprehensive email-validation specification, and it isn’t meant to be. It shows how we can replace an ambiguous requirement with explicit, observable expectations. We can make the judgment even clearer by specifying what counts as sufficient evidence. For example, the judge should identify the relevant validation logic and establish that invalid input can’t reach the persistence operation. If the evidence is insufficient, the criterion doesn’t pass.
We might also provide examples of valid and invalid input to clarify the rules. But examples should support the requirement, not replace it. Given only a handful of examples and no rule to anchor them, the judge might treat those examples as an exhaustive definition of correctness.
Notice, that we haven’t prescribed how the solution must implement validation. It might use a validation library or a regular expression, and it might validate input in a controller or a service. As long as it satisfies the stated requirements, those implementation details shouldn’t matter. It’s the same principle we apply when writing maintainable tests: test behavior rather than implementation details. A refactoring shouldn’t fail an evaluation merely because someone renamed a method or reorganized the code. Be precise about what success means, without unnecessarily prescribing how to achieve it.
Deterministic doesn’t mean accurate
If interpretation is the problem, why not eliminate it altogether? You could replace semantic judgments with deterministic checks. If a solution must use Contoso, check whether the generated code contains the string Contoso. Every run produces the same answer for the same input, so the check is completely deterministic. It’s also a terrible criterion.
The code might contain a comment saying Decided not to use Contoso, and the check would happily report success. Conversely, the solution might correctly use Contoso through an abstraction without ever mentioning that exact string in the inspected file. We’ve eliminated variability in the check without improving its ability to establish success. And the check quietly prescribes an implementation detail, the presence of one specific string, which is exactly what we decided to leave open.
Deterministic checks are excellent when the requirement itself is deterministic. Checking whether code compiles or tests pass doesn’t require an LLM. But when correctness depends on meaning, LLM judges offer something valuable: they can recognize semantically equivalent solutions despite differences in naming and structure. So the trick is to minimize interpretation of the requirements without eliminating the reasoning needed to evaluate the solution.
Challenge your rubric before trusting it
Writing an explicit criterion is a good start, but how do you know you’ve eliminated enough ambiguity? Start by reviewing the rubric critically. You can even ask an LLM to point out undefined terms and implicit assumptions, or places where a criterion supports more than one interpretation. A particularly useful question to ask yourself is: what do I assume the judge knows?
Then run the evaluation against a few representative solutions and have the judge provide evidence alongside each score. A pass or fail tells you what the judge concluded, while the supporting evidence helps you understand why. Without it, you’re left reconstructing the judgment yourself and possibly making incorrect assumptions about what happened.
With the evidence in hand, inspect the verdicts. Look for cases where the judge reaches an unexpected conclusion or relies on irrelevant evidence, and for requirements it interprets differently from what you intended. Refine the rubric where it allowed those readings, then run it again.
Also try running the rubric with a smaller, less capable model. A powerful model can sometimes compensate for missing instructions using its domain knowledge, which makes an ambiguous rubric appear reliable. A smaller model may expose those gaps sooner. Of course, not every failure indicates ambiguity. If you’ve made the requirements as explicit as reasonably possible and the smaller model still struggles, it may simply lack the capabilities the judgment requires. So rather than forcing every evaluation onto the smallest model, use it to learn how much model your rubric actually needs. That way, you don’t pay for a more capable judge merely to compensate for instructions you could have written more clearly.
Ambiguity has a maintenance cost
You might wonder whether all this effort is worth it. Why turn a short question into several detailed criteria when a capable LLM could probably figure out what you mean? Because that convenience comes with a hidden maintenance cost. If your rubric relies on the judge’s implicit domain knowledge, you need to establish that its judgments are reliable. And when you change the judge model, you need to verify that the new model makes sufficiently similar decisions.
Eventually, you’ll have to change models, because providers update and retire them. A rubric that works only because a particular model happens to interpret its vague instructions the way you intended is a fragile investment. Explicit rubrics aren’t immune to model changes, and they still need validation. But the less they depend on implicit knowledge, the easier they are to test, and the better your chances of running them on cheaper models.
Yes, a precise rubric might require fifteen criteria where you previously had one. And that’s okay. Once you’ve established that your rubric works, you can reuse it across many evaluation runs, and the upfront effort pays back through more reliable results and less time spent investigating questionable verdicts. You’re unlikely to eliminate every assumption, but you should make a deliberate effort to remove as many as you reasonably can.
The next time you write an evaluation criterion, don’t stop at asking whether the question has a yes-or-no answer. Ask what the judge needs to know to answer it and where that knowledge comes from. Then ask whether two reasonable judges could reach different conclusions. Your judge doesn’t know what you mean. It knows what you wrote. Make the difference as small as possible.