Your LLM Judge Is Just One More Model to Test

Most teams building agents spend weeks thinking carefully about prompts, retrieval pipelines, and model choices. The challenge is when it’s time to evaluate how these agents are performing for their intended use case. The low-effort route is to drop in the latest frontier model as a judge (using one LLM to grade another’s answer), write a generic rubric to score and take the result at face value. This is where a new blind spot creeps up: the LLM judge is a model too. It comes with the same failure modes, hallucinations, and tendency to confidently produce the wrong answer. That’s why there’s a pressing need to fine-tune evaluators and more broadly, optimize it for the domain you’re operating in.

The judge is the most surprising example, but it’s not the only one. Every part of your eval stack: the judge, the assertions, the rubric, even the prompts under test are knobs you continually tune, not something you write once and trust forever. This post walks through how we fine-tune the whole framework. The loop looks roughly like this: baseline run, triage from the trace, scoped fix, then regression and variance checks before you repeat.

Baseline → triage → scoped fix → regression. Every pass can touch a different part of the stack.

What an LLM Judge Is, and When to Reach for One

An LLM judge is one model grading another model’s output against a rubric. It stands in for a human reviewer, so you can score thousands of answers in seconds instead of relying on manual review.

Before LLM judges, many NLP tasks were evaluated with metrics like BLEU for machine translation, ROUGE for summarization, and exact match for closed-form question answering. BLEU measures how much a generated answer overlaps with a reference answer, mostly by matching short word sequences. ROUGE does a similar overlap check for summaries.

With LLMs, that approach may not be the most suitable as the output is open-ended, the wording can vary, and there may be many valid ways to answer the same question.

That is where LLM judges help. They can evaluate qualities that are hard to reduce to string matching, like whether an answer is relevant, grounded, and helpful.

Note: People often underestimate deterministic judges. If the behavior can be stated precisely (an exact value, a regex, a numeric range, a required field, or a passing test), use that first. It is faster, free, and stable across runs.

For cases that still need judgment, there are three common LLM-judge patterns:

• Direct scoring: grade a single answer against a rubric. Good for objective checks like faithfulness or instruction-following.
• Pairwise: show two answers and ask which is better. This is more stable, and it works well for subjective qualities like tone. It’s how we run our frontier-model comparisons.
• Reference-based: grade against a golden answer. Our assertions work this way: they list the facts any good answer should cover.

Another way to think about LLM judges is that they do not have to judge only one thing. A single response can be evaluated across multiple dimensions. Some examples are:

• Correctness: did the answer reach the right conclusion and cover the required assertions?
• Faithfulness: did the answer stay true to the source material, or drift into invention?
• Relevance: did it actually answer what was asked?
• Best answer: when comparing outputs from multiple models, which response is strongest overall?

When we say “quality,” we mean this whole set of dimensions, not a single pass/fail number. Looking at answers through multiple lenses helps catch cases where an answer is technically correct but still not the one you would ship.

The Problem with Off-the-Shelf Judges

So yes, LLM judges are useful. They make it possible to evaluate open-ended agent behavior at a scale human reviewers cannot match. But in enterprise evals, usefulness is not the same as reliability. The judge still needs context about the domain, the customer, and what a real failure looks like. Without that context, a few failure modes show up quickly:

• LLM judge tests phrasing, not substance: An uncalibrated judge penalizes an answer for different wording even when the meaning is identical.
• It can’t tell “recommends product X” from “recommends the right product.” A generic correctness rubric rewards a plausible recommendation as readily as the most correct one.
• It has no domain context. A general-purpose judge doesn’t know that multiband means something specific in your use case, or that violating frequency requirements is a serious error for certain stakeholders.
• It weighs every failure equally. Without a notion of severity, or failure modes, the score gives you no signal on where to actually improve the agent. We use traits to slice failures by capability so the score points at a real fix.

To make the numbers mean something, the fix is to treat the judge as a model and calibrate it appropriately.

Calibrating the Eval System

When a case fails under an LLM judge, it is not always the judge’s fault, and it is not always the agent’s fault either. The failure could be in the rubric, the expected answer, the reference data, or the way the judge was asked to read the case.

The tempting fix is to swap in a stronger model as the judge. That can help, but only up to a point. A model is only as good as the prompt and context you give it, so the whole eval system needs calibration.

Before tuning anything, name the moving pieces. For a single case, the judge is not just reading an answer in isolation. It is reading the query, the response, the rubric, the assertions, and any reference data you give it.

The different parts of an eval system: the query, rubric, and assertions the judge uses to score a response.

The Calibration Loop

The loop is simple, but it has to be repeated. Each step is doing one job: anchor the judge to human judgment, find where its reasoning diverges, and tighten the eval system until the score means what you think it means.

1. Label a small set of cases yourself. Start with a representative sample and decide what a good answer looks like for each one. Those human labels are what you are calibrating against. Without them, you cannot tell whether the judge is grading the way you would.

2. Run the judge on the same cases. Score that same set with the LLM judge, and keep the reasoning alongside the pass/fail. The score tells you the outcome; the reasoning is what you will use to figure out why the judge got it right or wrong.

3. Compare the judge’s decisions to your labels. Look at how often they agree. Raw agreement is a useful first check, but when most cases are passes, Cohen’s κ gives a clearer picture because it accounts for chance. If the judge disagrees with you on the cases that matter, the score is not ready to trust yet.

4. Inspect the disagreements. This is usually where the real work happens. Read the judge’s reasoning and ask what went wrong. Did the rubric miss a domain rule? Was the assertion testing wording instead of meaning? Was the reference data itself wrong? Each kind of disagreement points to a different fix: the rubric, the assertion, or the data, not automatically the agent.

5. Update the eval system. Use those findings to tighten the rubric, assertions, judge prompt, examples, or reference data. Swapping in a stronger judge model can help, but it should not be the only change you make.

6. Rerun, then keep those cases as regression tests. After each update, run the judge again on the same set and check whether agreement improved. Over time, that calibration set becomes the test suite for the judge itself. When someone later changes the rubric, the prompt, or the model, these cases show whether grading still holds.

What You Tune in Practice

The word “calibration” can sound abstract, but the actual work is concrete. You tune the parts of the eval system that shape how the judge reads the answer.

Assertions, Rubric, and Source Data

When the judge disagrees with a human label, the problem is often not the judge model. It could be the ground-truth assertions you generated for the case. It could be the source data behind those assertions: a wrong catalog value, a conflicting spec, a bad reference. Or it could be the rubric: too tight, too loose, or punishing the wrong kind of miss.

Say the agent recommends a valid product that meets the customer’s constraints, and a human would pass it. The judge fails it. Why?

• The assertion might be too rigid: it required one exact product name, even though another valid option was fine.
• The source data might be wrong: the catalog or reference behind the assertion has a bad value, so the case itself is testing the wrong thing.
• The rubric might be too tight: it treated a missing nice-to-have detail as a hard fail.

That is most of what calibration is. You inspect the disagreement, ask which of those three is off, and fix that before you touch the agent prompt. We go deeper on writing assertions in Designing Eval Cases.

Note: Even a well-tuned rubric does not remove judge bias. In pairwise comparisons, judges often favor the first answer they see, so we run both orderings and average. They also tend to reward longer answers, and sometimes prefer outputs from their own model family. A judge is non-deterministic too: borderline cases flip between runs, so we separate that grader noise from the agent’s own variance.

Triage Failures Before Tuning Prompts

Once the eval system has enough structure, the next discipline is triage: a failed case is not automatically a prompt bug.

Half the Failures Aren’t the Agent

In one recent run, of about 30 failing cases only around half were the agent actually getting something wrong; the rest split between bad data in the catalog and eval cases where the assertion or rubric was testing the wrong thing. If you treat every failed case as an agent bug, you spend most of your prompt-tuning effort chasing problems a prompt can’t fix.

Triage From the Trace

So we triage from the trace, not the raw score. In practice, this means reading the step-by-step record of which tools the agent called, in what order, and what came back. Then, place each failure in one of three buckets: the agent, the data, or the test (covered in Designing Eval Cases).

When a case fails and the evaluator’s reasoning looks off, the question is always the same: is the agent actually wrong, or is the assertion testing the wrong thing? Fixing a bad assertion is as important as fixing a bad prompt, and usually much cheaper. For the failures that really are the agent’s, the discipline that matters is scope: we start from a baseline run and change one section of the prompt, or one skill at a time. This gives a signal on whether the fix actually worked and where to look if something regresses. A change that fixes one failure mode and breaks three others isn’t progress, so after prompt or skill changes we run the eval on a large variety of queries as a regression measure and do a case-by-case analysis instead of just looking at whether the overall score went up.

Separate Signal From Variance

Going a step further, we run variance checks, because LLMs are non-deterministic and the same prompt on the same case can score differently across runs. In ours, roughly a third of individual cases flipped between pass and fail across identical runs, even while the overall score barely moved, so a single run’s per-case result can’t be trusted on its own. Before attributing an improvement to a change we made, we need to know whether the score delta is larger than the model’s natural variance, which we measure by running the same prompt twice on the same dataset. If a change produces a 4 percentage point (pp) improvement but the model naturally varies by ±3pp, we can’t claim it worked, and we need a second confirmation run to see if it holds. It’s slower, but it’s what separates a real signal from noise. This way, when we say something has improved, it’s more likely to hold in production.

What Compounds Over the Loop

Wiring up an LLM judge is straightforward. The hard part is the domain-specific judgment that accumulates around it: a rubric that knows what recommending the wrong product costs a customer, an assertion reconciled against a datasheet, a borderline case the judge now handles consistently.

None of that ships in the framework. It comes from the small, repeated corrections that tune the eval to how your customers actually work. That’s why pointing a frontier model at the problem eventually plateaus, while a well-tuned eval system keeps getting sharper the longer the loop runs.

When the judge is even slightly off, you can burn weeks tuning the agent toward the wrong thing. A good eval framework isn’t something you set up once and trust; it’s something you keep tuning until it understands your domain almost as well as you do.