Lore

LLM-as-Judge Validation

Overview

An LLM-as-judge eval is not trustworthy by default — it must itself be validated against human labels before a team relies on its verdicts. Skipping this validation risks a growing gap between judge output and ground truth that eventually destroys stakeholder trust in the whole evals effort.

Why "agreement" is a deceptive metric

Real-world failure rates are typically low, so a lazy judge that always predicts "no failure" can score very high on aggregate agreement while being useless — it just mirrors the base rate. The honest alternative is to disaggregate performance into true-positive rate and true-negative rate and inspect a confusion matrix, rather than reporting a single blended accuracy/agreement number.

Relationship to other practices

Judges are only worth building for problems identified through Error Analysis (Manual Trace Review) that are recurring and too costly to check any other way — see Eval Investment Triage. Judge outputs work best as binary pass/fail labels rather than continuous scores.

Meta-Evaluation: Measuring the Judge

Before trusting an LLM judge in production, validate it the same way you'd validate any classifier: compare its binary verdicts against human ground-truth labels on the same set of traces. Hamel Husain calls this "measuring the judge" — a judge is only as good as its agreement with a human labeler, so skipping this step means shipping an unvalidated classifier.

Human labels for this step come for free: they're the same labels produced during Open Coding and Axial Coding, no separate labeling effort required.

Tabulate the comparison as a confusion matrix (human label × judge label, both binary) rather than trusting a single aggregate score. From the matrix, compute:

Raw "agreement" (percent of traces where judge and human match) is a red-flag metric when failures are rare: a judge that always predicts "no failure" can score 90%+ agreement while catching zero real failures. Quote: "You can agree with it 90% of the time by just saying it never fails." Push for TPR/TNR instead of a single agreement number whenever someone reports "agreement" as judge quality.

See Binary Pass/Fail Evaluation for why the judge's own output should be binary in the first place, and Eval Investment Triage for when a validated judge is worth building versus a Code-Based (Reference-Based) Evaluation.

Few-Shot Examples and the Held-Out Set

Adding few-shot labeled examples to a judge's prompt can improve its accuracy, but if the same traces used as few-shot examples are also used to measure the judge's TPR/TNR, the resulting accuracy is inflated — the judge is effectively being graded on its own training data.

Split labeled traces into two disjoint sets: one used as few-shot examples inside the judge prompt, and a separate held-out set used only to compute TPR/TNR. Only the held-out numbers should be trusted as the judge's real accuracy.

CI and Production Monitoring

Once a judge clears its TPR/TNR bar on the held-out set, deploy it in two places:

This two-pronged deployment turns a one-off validation exercise into a standing eval, and pairs with Eval Investment Triage's guidance on which problems are worth this level of investment.

Class-Imbalance Trap in Judge Agreement

A judge validated purely on raw agreement-with-human percentage can look deceptively good when failures are rare. If the true failure rate is roughly 10-15%, a judge that simply never flags any failure ('everything passes') will still agree with the human label on the ~85-90% of traces that actually did pass — producing roughly 90% agreement while being completely useless at the one thing it exists to do: catch failures.

This is why validating a judge needs to look at true positive rate and true negative rate separately rather than a single blended agreement score, which class imbalance can make arbitrarily misleading.

Ground Truth From Axial Coding

Calibrating a judge requires human-labeled ground truth, but this doesn't require a separate labeling pass: it falls out for free from the categories already produced during Axial Coding on the Error Analysis (Manual Trace Review) traces. The pass/fail and category labels a reviewer assigns while doing manual trace review become the held-out ground truth set the judge is checked against.

Few-Shot Overfitting Risk

Adding few-shot examples to a judge prompt without splitting the labeled data into a training set and a held-out set lets the judge memorize rather than generalize: it can score 100% agreement simply because it's recognizing traces it already saw in its own prompt ('If I put all of these traces in my prompt, it would get 100%'). Judge validation numbers are only meaningful when computed on traces the judge's prompt has never seen, mirroring standard train/test separation.

Cost of Skipping Validation

Hamel Husain and Shreya Shankar argue that skipping validation of an LLM judge against human judgment is precisely how teams end up with evals that don't reflect reality — and that this failure erodes not just trust in the eval itself but in the credibility of the person who built it. They trace much of the visible 'evals don't work' backlash (e.g., on X) to teams who shipped unvalidated judges, often naive Likert-scale scorers, rather than to evals being inherently unreliable; they also note some of the debate is a definitional disagreement about what 'eval' even means.

Scoped Judge Prompts

Each LLM-as-judge should be scoped to one narrow, specific failure mode with an explicit set of trigger conditions, not a general-purpose quality check. It should return a binary pass/fail verdict rather than a scale, since scales let evaluators dodge the decision and produce averages that are hard to interpret. The same judge prompt is reused in two places: as a unit test in CI, and as a continuous monitor — see Feedback-Loop Pipeline: Real-World Data to Evals — on sampled production traffic.

Confusion-Matrix Validation

Before trusting a judge, build a confusion matrix — a pivot table of human/axial-code labels against judge labels — rather than relying on a single aggregate agreement percentage, which can look deceptively high for rare failure modes (a judge that always says "pass" scores well on agreement if the failure is rare). Iterate on the judge prompt until the off-diagonal (disagreement) cells shrink. This is the same counting discipline as Basic Counting / Pivot-Table Frequency Analysis, applied to judge reliability instead of raw failure-mode frequency.

Agreement Percentage Is a Dangerous Metric

Direct warning from the source: 'you can easily have 90% agreement by just having a judge say... it passes all the time' — a judge that always outputs 'pass' scores roughly 90% raw agreement with human labels when the true error rate is only about 10%, because agreement is dominated by the easy majority class. Bare agreement percentage, used alone, is called out by name as a 'dangerous metric' for validating a judge — it must be paired with class-aware measures (e.g. recall on the failing cases) or it hides exactly the errors the judge exists to catch.

Judge Built From an Accumulated Dataset (Linear)

Linear builds its judge model/prompt from an accumulated dataset of real agent traces rather than a hand-written rubric alone, and uses it to score subjective qualities such as whether the agent extracted the right information from a ticket or conversation. Because the judge itself can be wrong, the team manually spot-checks the judge's own accuracy rather than trusting its scores unconditionally. Used alongside Code-Based (Reference-Based) Evaluation as the subjective half of a mixed eval suite; see consistency-scoped-eval-principle for when this kind of check is worth writing at all.