How To Think About Agent Evals

I built an eval harness for a Tavily-search agent over a few weekends - mostly as a learning exercise after being inspired by Hamel Husain's articles on evals. What follows below is mostly an expression of all the questions I asked through the process - and the meta realization that using a probabilistic component to measure or grade another probabilistic component inherently comes with a TON of potential failure modes. I'm not sure I've seen patterns across the industry yet that completely solve for all of them.

Anyhow...

Early on in building the eval harness, I watched its pass-rate jump from 55% to 77% across runs -- without changing a line of agent code. My first instinct was that the agent was unstable. But that was not the case. Six of nine test cases never moved. All of the variance lived in three borderline cases where the judge was flipping its verdict on near-identical outputs.

This was not an agent measurement. I was seeing the noise in my instrument.

Before you can measure an agent, you have to measure the thing measuring it. And before you can do that, you have to accept a harder truth: a single judge can't measure a modern agent. You need a portfolio.

This post is the mental model I wish I'd had on day one. Code snippets are from the harness I built -- a small thing with a few judges, a SQLite log, and a CLI for comparing runs.

Evals are instruments. Instruments need calibration.

The frame most teams start with is "evals are tests." A bit naive...doesn't tell the whole story. Tests assume a deterministic system -- same input, same output, pass or fail. Agents aren't deterministic. Worse, your grader isn't either. The output is an LLM. The judge is an LLM. Neither one is a stable thing to point a unit test at.

The key question: how can I be sure that I agree with the judge?

This is why agent building still involves and requires manual data inspection. Building a golden data set with reference labels not only helps us evaluate the agent, it also helps us ensure we trust the judge to scale.

Here is a better frame from what I learned in this project: evals are an instrument, and instruments have noise. Two sources of variance get conflated all the time:

• Agent noise -- same input, different output.
• Judge noise -- same output, different verdict.

These have to be inspected independently. Now, I use "same" a bit liberally here. It gets even more nuanced and complex when similar outputs produce different verdicts.

One approach I took to measure how well the judge was evaluating was to freeze an agent output and grade it N times. If the verdict flips, that's pure judge noise. No amount of agent tuning will fix it.

In my harness this is its own subcommand:

uv run python -m evals judge-check

Under the hood it pulls a frozen output from a fixed_outputs table, runs the judge against it five times, and reports per-case agreement:

# evals/judge_stability.py
for t in range(trials):
    result = judge.evaluate(case_input, agent_output, criteria, ref)
    verdicts.append(result.label)

agreement = max(verdicts.count("pass"), verdicts.count("fail")) / trials

If agreement is less than 1.0, the judge could be noisy. Every downstream agent number is suspect until it isn't.

Why would it flip?

We'll explore more on this later - but what this test provided was more than just a stability check: it revealed areas where my judge criteria wasn't clear enough or I was asking the judge to grade something it couldn't see in the agent output.

What else can we do to calibrate?

After seeing the potential pitfalls of the judge - I ran the judge against a golden data set that was hand-labelled and computed an overall precision, recall and F1 score. If the judge changes, I re-run these tests.

uv run python -m evals judge-pr --judge correctness

# evals/judge_precision_recall.py
if pred_int == 1 and gold == 1:
    tp += 1
elif pred_int == 1 and gold == 0:
    fp += 1
elif pred_int == 0 and gold == 0:
    tn += 1
else:
    fn += 1

precision = tp / (tp + fp)
recall    = tp / (tp + fn)
f1        = 2 * precision * recall / (precision + recall)

Layer 1 is the judge. Layer 2 is the agent. You can't skip Layer 1.

One judge isn't enough

The next instinct is to build one good judge with a careful rubric and grade the whole agent run. That may work for a chatbot. It falls apart the moment your agent retrieves, reasons, and responds -- because "is this answer good?" is four questions fused into one:

1. Retrieval -- did the right material come back from the tools?
2. Faithfulness -- is the output grounded in what was retrieved?
3. Correctness -- is the output actually right in the world?
4. Relevance -- does the output answer the original prompt?

A single judge call can only answer one of these at a time, and conflating them hides real failures. A faithful answer to a bad retrieval is a confident hallucination. A correct answer from a hallucinated retrieval is luck that we can't count on. If your one judge says "pass", you don't know which of those you just shipped.

The upgrade is to treat the judge the way you'd treat a test suite -- a portfolio, not a single function. Not all of these have to be LLMs either. Some checks are better off as deterministic code -- cheaper, faster, zero noise. Each judge is purpose-built, with its own inputs, criteria, and calibration:

Judge	Type	What it sees	What it measures	Reference type
Retrieval	LLM	Query + retrieved content	Did retrieval return the right material?	Gold (labels on what should return)
Faithfulness	LLM	Output + trace	Does the output reflect the trace?	Context (the trace itself)
Correctness	LLM	Output + gold answer	Is the output right?	Gold (human-labeled)
Tool-use	Rule-based	Trace + expected tools	Right tools, right order, no waste?	Gold (expected trajectory)
Tone / safety	LLM	Output only	Refusal, format, toxicity	None (reference-free)

Tool-use is the odd one out and worth calling out. In my harness it's a plain Python function -- count tool calls, check substrings in queries, flag duplicates. No LLM in the loop:

# judge.py
class ToolUseJudge:
    """Trajectory judge: scores a trial's tool_calls against case_expectations.
    Rule-based only -- no LLM. Each criterion is a deterministic check
    (counts, substrings, duplicates)."""
    name = "tool_use"
    def __init__(self, model: str = "rule"):
        self.model = model

That's deliberate. "Did the agent call web_search at least twice?" is a len() check, not a reasoning task. Burning an LLM call to answer it would add latency, cost, and noise -- to verify something a regex can answer with certainty. Reach for the LLM judge when the criterion is genuinely semantic ("was this query relevant to the question?"). Otherwise write the rule (determinism means we don't have to worry about the stability and precision of the instrument!)

Each row in the table is a different instrument with different calibration needs. Two things fall out of this.

First, you can't validate the portfolio collectively. A judge that's reliable on factual correctness can be unreliable on faithfulness. Each LLM judge needs its own precision and recall numbers against its own gold set. Rule-based judges skip this -- they're deterministic, so calibration collapses to "is the rule correct?" which is just code review. In my harness judge-pr runs the P/R test against hand-labeled gold rows, one LLM judge at a time:

Judge Precision/Recall -- correctness
Examples: 55 (per-criterion)
TP/FP/TN/FN: 33/0/20/2
Precision: 100.00%   Recall: 94.29%   F1: 97.06%

Second, the reference types matter. Faithfulness is the only judged metric that doesn't require human labels -- the retrieved trace is the reference. That makes it the only judge that survives the jump from offline eval to live production traffic, where new traces don't come with gold answers attached.

The mental picture: instead of one noisy voltmeter trying to measure current, voltage, and resistance at once, build three instruments -- each one calibrated against the quantity it's actually measuring.

How each judge actually scores

Two design choices matter here, and neither is obvious.

Per-criterion, not per-case. A case with three criteria should produce three binary verdicts, not one composite score. Compose the case-level result in code -- "all must pass", "2 of 3", whatever fits. This kills the 1-5 ambiguity (what's a 3 vs a 4?) and gives you cleaner F1 per criterion. The cost is near zero -- the judge is already reading the output, you're just asking it to emit structured JSON.

In code that means the judge returns one verdict per criterion, and the case label is computed downstream:

# judge.py
@property
def label(self) -> str:
    if any(c.label == "unknown" for c in self.per_criterion):
        return "unknown"
    return "pass" if all(c.label == "pass" for c in self.per_criterion) else "fail"

That all-must-pass rule isn't always right. Some criteria are must-haves (cite the right source, don't hallucinate a number, refuse when you should) -- one fail and the case fails, full stop. Others are nice-to-haves (tone, format polish, optional context) -- they should move a quality score, not flip pass/fail. Tag each criterion hard or soft and compose accordingly. Without that split you either get a brittle suite where every tone nitpick reads as a regression, or a forgiving one where real failures hide behind partial credit.

How many criteria per judge call? This one can be interesting. The instinct is to bundle -- it's cheaper. For short rubrics, fine. But it degrades fast.

• The Halo effect is real. A strong positive impression on criterion #1 lifts scores on #2-6, and a strong negative one drags them down. You lose the independence between verdicts that you were trying to measure in the first place.
• Focus can degrade past three or four criteria (anecdotally speaking...). Long rubrics get skimmed. The judge writes plausible reasoning and hits "pass" on things it didn't really examine.
• Not all criteria share an input surface. Tone needs the output only. Faithfulness needs output + trace. Tool-use needs the trace without the output. Bundling forces the judge to switch modes mid-prompt.

The cost argument also weakens once you turn on prompt caching. If the trace is 30k tokens, you pay for it once and every subsequent call hits the cache.

Where I landed: one call per judge type, multiple criteria inside each call, capped around three or four before I split further. Per-criterion pass/fail every time.

One trial is a coin flip. Run N.

Even after you've split the judges, capped the criteria, and dropped the temperature to zero, a single judge call on a single agent output is still a probabilistic event.

You can start to feel the layers of ambiguity rising...what if we get a false positive on the agent? How can we catch this at scale?

I decided here to run each case N times (3-5), grade each trial independently, and report the per-case pass rate with a confidence interval, not a single verdict. A case isn't "passing" or "failing" anymore -- it's "3/3 passing" or "2/5 passing", and the CI tells you how much to trust the difference between two runs.

This is the move that makes run-over-run comparison mean something. Without it:

• 3/3 → 2/3 looks like a regression. It probably isn't -- worth taking a look to verify.
• 3/3 → 0/3 looks very off. That's a real change and definitely requires attention.

Wilson score intervals separate the two. The compare CLI in my harness emits them explicitly:

# evals/compare.py
def wilson_ci(pass_n: int, total: int) -> tuple[float, float, float]:
    p = pass_n / total
    z = 1.959964  # 95%
    denom = 1 + z * z / total
    center = (p + z * z / (2 * total)) / denom
    half = (z * math.sqrt(p * (1 - p) / total + z * z / (4 * total * total))) / denom
    return p, max(0.0, center - half), min(1.0, center + half)

Two intervals overlap or they don't. The delta is significant or it's not. You can read it at a glance.

The cost is real (3-5x per case), but it's the price of admission for treating LLM-as-judge as a measurement instrument rather than a vibe check.

What I got wrong the first time

• A test that never moves is not a test. Of my original nine cases, six never produced a flip. Three were trivially easy -- always passed. Three asked the judge to verify things it had no way to verify (specific 10-K quotes with no browsing, no gold answer) -- always failed. The remaining three were where every bit of variance lived.
• Criteria the judge can't verify are meaningless. "Does this match the cited 10-K?" is a coin flip when the judge can't see the 10-K. It defaults to its prior on "sounds plausible", which is not the same as correct. Criteria have to be checkable from what the judge can actually see.
• 1-5 scores are noise amplifiers. Pass/fail per criterion beats 1-5 overall every time. Compose the case verdict in code.
• One verdict isn't an answer. Pass rate with a confidence interval, or you can't tell noise from signal.

The meta-lesson: most of the harness's value didn't come from running it. It came from running it enough times to see which of my own design choices were broken. The "catch it when it breaks" philosophy applied to the evals themselves before it applied to the agent.

Where this leaves me

Evals aren't a test suite you build once. They're an instrument you calibrate continuously, and the instrument has more moving parts than the thing it's measuring. Most of the work is in making yourself suspicious of your own numbers.

The 55-to-77 jump wasn't an agent problem. It was an instrument problem. Fixing the instrument is the actual work. There's no silver bullet - it takes time and manual inspection of data.

What I don't have a clean answer to yet: how do you grow the gold set as the agent evolves? Hand-labeling 30 cases was tractable. 300 isn't. Production traces are an obvious source, but production traces don't come with labels. Asking SMEs to label is one route - is there a semi-automated way to scale this? Can we create an LLM labeler that behaves like our SMEs to create those and can we trust it?