When building evals for AI coding agents, string contains is incredibly tempting. It’s fast and deterministic. And it costs practically nothing, so you can run it on every change in CI without worrying about inference cost or latency. But is using it making your eval lie to you?

Want to know whether the agent used Azure? Look for Azure, right? Want to know whether it used a particular API? Look for the API name. Want to know whether it created the right configuration? Check whether the expected string is in the file. Pass. Ship it. But you haven’t established any of those things.

What did your grader prove?

Suppose your eval says that the generated solution must use Azure, and your grader checks whether the generated files contain the string Azure. The grader passes. What do you now know? Only that the string Azure appears somewhere in the generated files. That’s it.

The string could appear in code that uses Azure, but it could also appear in a comment. After all, a single line is enough to satisfy your grader. Here is all it needs to see:

// Don't use Azure here.

That comment passes the check. So would dead code, an unused dependency, documentation, or a test fixture. A configuration file could contain the string even though the application never loads that file. Your grader finds what it’s looking for, but the behavior you care about is still absent.

The same problem works in reverse. If the string isn’t present, can you conclude that the solution doesn’t use Azure? No. The agent could use an SDK without mentioning Azure explicitly, refer only to a package or class name, or express the same intent using words you didn’t anticipate. Natural language makes this worse because the difference between did and didn’t can be one word while the keyword remains exactly the same. A deterministic grader gives you a deterministic answer, but that answer only means what the check establishes.

Finish the sentence

There’s a simple test I like for eval criteria. Complete one sentence for a passing result, then another for a failure. Be literal:

If this grader passes, I now know that ______.

If this grader fails, I now know that ______.

For a string contains "Azure" grader, the answers are narrow. They describe only what the grader observed. Anything more is an inference:

If this grader passes, I know the string Azure occurs in the inspected content.

If this grader fails, I know the string Azure doesn’t occur in the inspected content.

If that’s what you wanted to measure, great. But when you complete the first sentence with the agent correctly used Azure, you’ve made a leap your evidence doesn’t support. The grader isn’t necessarily inaccurate. It might be perfectly accurate at what it does, while we ask it to prove something it can’t.

Easy evidence can give you false assurance

It’s easy to see why people use simple deterministic graders. Agentic evals cost money and can take a long time to run. LLM judges add inference cost and latency. Building, running, or deploying generated applications adds infrastructure and complexity. In comparison, string contains takes milliseconds, which makes it ideal for CI… and potentially terrible evidence.

So you start optimizing an eval around what’s convenient to measure instead of what you need to know. The result can be a fast, repeatable evaluation pipeline that gives you false assurance on every run. I’d rather run fewer evals that produce meaningful evidence than continuously run evals that confidently tell me very little.

Ask the system that can answer the question

If you want to know whether generated code builds, build it. We’ve seen solutions receive perfect scores from an LLM judge only to fail compilation. The judge could assess whether the implementation looked correct, but only the compiler could verify it.

If you want to establish whether dependencies can be restored, restore them. If you want to know whether a JSON file follows a schema, validate it. If you want to establish whether tests pass, run them. And where reasonably possible, if you want to establish whether the application works, run it.

This is where deterministic graders work well because the check directly establishes the property you care about. Using an LLM to predict whether a project will compile makes little sense when the compiler can answer authoritatively. You’d be asking an LLM to approximate an answer that an existing tool can give you.

But running generated applications is harder than building them. Real applications need credentials and configuration. They also need data, infrastructure, and access to external services. Deploying them adds another layer of complexity. Unfortunately, coding agents often struggle at those boundaries, which makes it tempting to verify something easier instead.

So instead, you check for the expected artifacts or words in the output. An LLM judge says the implementation looks plausible. Then you treat those observations as evidence that the integration works, even though none of them exercised it. Sometimes running the complete solution genuinely isn’t practical, and that’s okay. An eval doesn’t need to prove everything, but its report should make clear what it did and didn’t establish.

LLM judges answer different questions

Semantic criteria are different. An LLM judge can distinguish use Azure for storing the data from do not use Azure for storing the data. It can inspect surrounding code, reason about how components relate, and recognize implementations that don’t contain the vocabulary you anticipated when writing the eval.

But an LLM judge doesn’t eliminate the evidence problem. It can’t reliably replace building and running the software. And even if it could reproduce every relevant compiler, package manager, test runner, or runtime check, why would you ask it to? We already have those tools. Use LLM judges for semantic questions and deterministic tools when they can answer a question directly.

Treat agentic evals like integration tests

When you evaluate an MCP server, skill, extension, or another capability for a coding agent, you care whether the agent can complete the task successfully. Finding a particular implementation detail in its output tells you far less. That’s why I find it more useful to think about these evals as integration tests than unit tests.

In our evals, building, testing, running, and deploying are separate gates. Alongside them, LLM judges evaluate the semantic properties relevant to each scenario. Keeping the checks separate shows you whether dependency restoration failed, compilation broke, or a buildable implementation missed an important requirement. One broad proxy shouldn’t have to answer all of these questions.

Still, deterministic graders are excellent when the property you’re measuring is deterministic. The problem starts when you choose a convenient signal and quietly expand the claim you make from it. Before adding a grader to your next agentic eval, finish these two sentences:

If this grader passes, I now know that ______.

If this grader fails, I now know that ______.

Then compare those answers with what you intend to claim about your agent, skill, MCP server, or extension. If they don’t match, find stronger evidence. Your eval doesn’t become trustworthy because it runs on every commit. It becomes trustworthy when its evidence supports the claims you make from it.