LLM judges made automated AI evaluation practical, and introduced new ways to fool yourself. Here's how I design judge panels, and what I'd ask before trusting one.

At Imagine Learning, I designed a math-tutoring chatbot that failed to meet the business's quality requirements. My conclusion was that the technology was too experimental at the time, and that finding helped redirect the effort toward simpler harassment detection, which fit the original business need better. I believe that direction did meet the original design goals better. However, as I've gained more experience with conversational AI, I've realized that my perspective only covered half of the question.

It told us the model couldn't meet the bar on its own. It didn't ask what design could meet the bar with the model's help: offering a teacher three candidate responses instead of one, say, or resolving units deterministically instead of generating them. The most useful evaluations don't stop at "not good enough." They also say what the system would have to look like to pass.

The state of the art in language models then was the bidirectional LSTM, and attention mechanisms were brand new. When I tested models with attention against the benchmarks we'd set, they really did produce better results.

Measuring "better" was the hard part. We measured with the standard metrics of the time, plus people. BLEU counts overlapping words and phrases with a reference answer. Cosine similarity between word embeddings such as word2vec scores how close an answer's words are to the reference's. Both told you whether the expected vocabulary was showing up. They told you very little about whether an answer was right. For that, you needed a human evaluator, and human evaluation is slow. That made it nearly impossible to iterate quickly.

Today a language model can grade another model's answers. That makes it possible to try dozens of prompt or retrieval variants in an afternoon and keep the one that scores best. It's also a new place for an AI system to fool the people who built it, because the judge is a model too.

I've scored models against ground truth with LLM judges and tools like Arize, and built agent systems whose output has to pass judge scoring and regression gates.

Don't ask one judge for one number

A common design is a single prompt: "Rate this answer from 1 to 10." That produces a number, but the number blends several separate questions. Is the answer correct? Is it supported by the sources? Is it complete? Is it readable? A 7 tells you nothing about which of those failed.

I build a panel instead, with each judge asking one specific question. For an article, "good" means more than accurate sentences. The piece has to give its intended reader something useful, develop its argument coherently, support its claims and follow the requested format. The judges need the editorial brief, the draft and the relevant sources to assess those properties.

Consider an article comparing two approaches for a technology decision. Its panel could look like this:

Judge What it checks How it can fail
Content usefulness Does the article help the intended reader make the decision in the brief, with concrete examples and reasons for its recommendations? Generic advice can sound useful unless the audience and decision are explicit.
Completeness Does it cover the required questions, compare both approaches on the requested dimensions and explain the tradeoffs and limits? Mentioning every topic can pass a shallow check even when a key comparison is never developed.
Writing quality Does each paragraph advance a coherent argument, with precise wording, clear transitions and terms the reader can understand? Polished prose can conceal a contradiction, an unexplained term or a conclusion that doesn't follow.
Citation support Does each claim needing evidence have a citation whose specific passage supports that claim, including its scope and qualifications? A reputable source on the same topic can be mistaken for support. A result from one benchmark can become an "always" claim.
Fact check Do numbers, dates and factual assertions agree with authoritative records, and does the arithmetic work? A citation can accurately represent an outdated or mistaken source. Sharing the writer's retrieval path also preserves its blind spots.
Structure and format Does the draft follow the requested section order, heading hierarchy, length, table layout and citation style? An attractive layout can get a high score while omitting a required section or breaking a footnote.

Require evidence with each verdict: the passage being judged, the criterion it meets or fails, and a short explanation. A completeness judge should name the missing requirement and where the draft stops short. A writing judge should point to the sentence whose wording or logic needs repair. A citation judge should pair the article's claim with the supporting source passage and its page or section. If the source cannot be inspected, mark the claim unverified.

Use objective checks where they fit: recompute arithmetic, count words, check required headings and validate Markdown tables, links and footnote references. A working link establishes that a source is reachable; the citation judge still has to assess whether its contents support the associated claim. The structure judge can then assess whether the sections actually do the work the brief requires.

Keep the verdicts separate. An article might be useful, beautifully written and correctly formatted while its central claim has an unsupported citation. A weighted average can still make that draft look strong. Treat unsupported material claims as a reason to revise or seek review before acceptance, regardless of the prose score. Narrow judges make it easier to see what failed and what needs to change.

Define good. Then judge. Develop criteria for the task you actually care about: start with the outcome, develop task-specific criteria, calibrate against experts, and refine and re-test. For article evaluation, ask whether the cited passage supports the claim.
Define good. Then judge. Develop criteria for the task you actually care about, calibrate against experts, and refine using disagreement cases.

Watch the fact-checker's sources

The fact-check row deserves its own section, because it's the easiest judge to fool.

Suppose your system answers questions from a document index, and your fact-checking judge verifies answers by looking things up in the same index. If retrieval misses a document, the answering system never sees it, and neither does the judge. The answer is wrong, the judge agrees, and the score looks fine. The two failures are correlated.

A fact-check judge is only as independent as its evidence. Where it matters, give it a different path to the truth: a structured database, a curated reference set, or the gold-standard answers your subject-matter experts wrote.

Calibrate the judges against experts, then keep calibrating

A judge is a measurement instrument, and instruments need calibration. The process I recommend:

  1. Build a gold set. Subject-matter experts select or write reference answers for representative questions, including hard and unanswerable ones. For articles, have human editors and subject-matter reviewers assess drafts against the same brief and sources, recording verdicts and passage-level reasons for each criterion. Include strong drafts, borderline ones and polished drafts with missing coverage, misleading citations or format violations. Resolve disagreements about the rubric before using those judgments as a reference.
  2. Measure agreement. Run each judge on the gold set and compare its verdicts with the experts'. Check each criterion separately, especially failures the judge incorrectly passes. An editorial panel that agrees on style but misses unsupported claims needs work. Look at the passages behind disagreements, not just the overall agreement rate.
  3. Fix the judge before you trust it. Rewrite criteria, add examples or change the evidence it sees, then measure again.
  4. Only then automate iteration. Use the calibrated panel to compare prompt and retrieval variants quickly. Agree on acceptance requirements in advance; rank variants by a weighted score only after they meet those requirements.
  5. Re-check the winners with people. Confirm that the top-scoring variant is actually better on the gold set. For articles, ask whether editors would accept the draft and whether their reasons match the panel's. If the judges and the experts drift apart, recalibrate.

Zheng and colleagues found that strong LLM judges agreed with human preferences more than 80% of the time, excluding ties, which is about as often as humans agree with each other. They also documented systematic biases: favoring the answer shown in a particular position, and favoring longer answers.1 Later work found that LLM evaluators can recognize their own outputs and score them higher than equivalent text from other sources.2

The trap in automated iteration

Once a panel is in place, it's tempting to let an optimizer try hundreds of prompt variants and keep whatever scores highest. That works, up to a point. Push it far enough and you start optimizing for the judges rather than for the users: answers that are longer, more confident or phrased the way the judge likes, without being more correct.

Two safeguards keep this honest. Hold out part of the gold set so that it's never used during iteration, and score the final candidates on it. And keep a human review of a sample of the winners. An improvement that experts can't see in the output isn't an improvement.

What I'd ask before trusting a team's evaluation numbers

When a vendor or an internal team reports "our judge scores 94% accuracy," these are my questions:

  1. What exactly does each judge measure? One blended score is a warning sign.
  2. How often does the judge agree with subject-matter experts, on how many cases, and where does it disagree?
  3. Does the fact-check judge use evidence independent of the system's own retrieval?
  4. Is the judge from the same model family as the system it grades? Judges have been shown to favor their own outputs.
  5. Are answer positions randomized and length controlled for in comparisons?
  6. Was the final result measured on held-out cases that played no part in tuning?
  7. When did someone last check the judges against people?

Ask for the disagreement cases, the examples where the judge and the experts parted ways. They tell you more about an evaluation than its headline score does.


Sources and notes

  1. Lianmin Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS Datasets and Benchmarks Track. BLEU is described in Kishore Papineni et al. (2002), BLEU: a Method for Automatic Evaluation of Machine Translation, ACL.

    Return to text
  2. Arjun Panickssery, Samuel R. Bowman and Shi Feng (2024), LLM Evaluators Recognize and Favor Their Own Generations.

    Return to text