When you evaluate an AI coding agent, it’s tempting to focus on one number: the score. Say you’re evaluating the base behavior of an agent and it passes 80% of your criteria. You add a skill, run the eval again, and now it passes 95%. Success. Except the first run cost $0.34 and the second cost $5.26. You improved the score by 15 percentage points and increased the inference cost more than 15 times. Is that still a success?

Quality is only one dimension

An eval score tells you how well the agent performed against the criteria you’ve defined. But whether the result was worth getting depends on what it cost. When you’re evaluating a change to an agent, such as adding a skill, changing instructions, switching models, or exposing a new tool, look at two dimensions: quality and inference cost.

Think of them as two axes. High quality at low cost is the obvious sweet spot. Low quality at high cost is something you want to avoid. The other two combinations are where the decision becomes interesting.

High quality at high cost might be exactly what you need for one task and completely unreasonable for another. Lower quality at dramatically lower cost might be an attractive tradeoff, provided you understand what’s missing and can live with it. Your eval provides the evidence, but you still need an opinion to make the decision.

There is no universally acceptable cost for an agent task because what makes sense depends on what the task is worth. A price of $5.26 can be reasonable while $0.34 can still be wasteful. You need to know what you’re buying for that money.

The same applies to quality. Suppose your agent scores 87%. Is that good? You can’t tell from the number. If the agent solves the hardest parts of the task and leaves a few trivial things unfinished, 87% might be incredibly useful. If it handles all the easy work but consistently fails the one difficult requirement that made you reach for an agent in the first place, 87% might be useless.

If your eval is based on meaningful criteria, use the aggregate score to compare configurations, then inspect which criteria passed and which didn’t. It’ll help you understand if the agent handles what you care about. A useful decision combines that knowledge with what you’re willing to pay for it.

Don’t measure one run

Agent behavior is nondeterministic. Run the same task several times and you might get different results and very different costs. Even consistent scores can hide an unpredictable bill. Imagine five runs all scoring around 95%, with these inference costs:

  • $0.52
  • $0.61
  • $0.73
  • $1.84
  • $5.12

Reporting the average hides how widely those costs spread. Your quality might be predictable while your cost varies by almost ten times. Someone eventually has to pick up the bill, and businesses don’t have unlimited budgets.

The reverse is equally concerning. Stable cost with wildly varying quality leaves you unsure what that money will buy. Ideally, repeated runs cluster in the part of the quality-cost space that’s acceptable for your task. When they don’t, the spread itself is a finding.

When you see unexpected variance in quality, cost, or both, look at the trajectories. They can reveal whether an expensive run explored several approaches, read far more context, repeatedly called a tool, or missed the right path at first. That gives you a cause to investigate even when the final result hides it.

Cost compounds

During development, inference costs can feel insignificant. What’s another few dollars? But you’re evaluating something that might eventually perform the task repeatedly. Going from $0.34 to $5.26 means spending an extra $4.92 every time the task runs. At 100 runs, that’s $492. At 10,000 runs, that’s $49,200. Let that sink in.

How much that matters depends, again, on the task. Something that runs twice a year has very different economics from something that runs thousands of times every day. Expected volume is part of deciding what you’re willing to pay.

Set your boundaries

Ideally, before running your evals, you have some idea of your acceptable quality and cost. You don’t need to get it exactly right. Start with an opinion. Perhaps $100 for scaffolding a project is obviously too much. Perhaps getting below 90% quality makes the result create more work than it saves. Those boundaries can become more precise as you collect data and learn what the agent can actually do.

What’s dangerous is having no boundaries at all. When someone else is picking up the inference bill, it’s easy to behave as if the budget is unlimited. Then every quality improvement looks attractive regardless of what it costs.

Eventually, someone will look at the bill. So record the quality and inference cost of every run. Run the eval enough times to see their variance. Look at which criteria are responsible for quality gaps, then inspect trajectories when either quality or cost behaves unexpectedly.

Finally, consider how often the task will run and develop an opinion about what you’re willing to pay for the quality you’re getting. Optimizing for either the highest score or the lowest cost ignores half of that decision. You’re looking for a quality-cost tradeoff that makes sense for your task, with enough consistency that you can rely on it.

A useful eval records the whole tradeoff. The score is only half the result.