Skip to content

Guide library

Guide · Quality, cost & reliability

LLM-as-a-Judge: How to Build a Grader You Can Trust

LLM judges make open-ended evals scalable, but they favour first-listed answers, longer answers and their own writing. Here is how to pick a judge format, write a rubric, and check a judge against human labels.

Metarix Editorial · Nov 10, 2026 · Updated Nov 15, 2026 · 10 min read

Promptfoo documentation card titled LLM Rubric, the LLM-as-a-judge grader in Promptfoo

An LLM judge is a model that grades another model's output against a rubric. It is the fastest way to score open-ended answers, such as summaries, support replies or agent explanations, when exact-match checks can't tell good from bad. It is also easy to get wrong: judges favour the first answer they see, longer answers and their own writing.

This guide covers when to use a judge at all, which of the three judge formats to pick, the biases documented in research, how to write a rubric a model can follow, and how to check your judge against human labels before you trust its scores.

When to use a judge, and when not to

Start with the cheapest grader that can actually tell right from wrong. Anthropic's evaluation guide ranks code-based grading as the fastest and most reliable, human grading as the most flexible but slow and expensive, and LLM-based grading as fast and flexible but something to "test to ensure reliability first, then scale." OpenAI's graders guide gives the same order: use an LLM judge "when code falls short."

Scroll the table horizontally on smaller screens.

Choosing a grading method
MethodGood forWeak atExample
Code checksExact answers, JSON shape, tool calls, tests that pass or failAnything that needs judgmentString match, schema validation, unit tests
LLM judgeOpen-ended quality: tone, completeness, faithfulness to a sourceMath and multi-step reasoning; can drift or be biasedRubric grader, pairwise comparison
Human reviewGround truth, new tasks, high-stakes callsSpeed, cost, consistency between reviewersBlind A/B review, expert scoring
Summary by Metarix from Anthropic and OpenAI evaluation docs, accessed Nov 10, 2026.

In practice, many eval suites mix all three: code checks for everything that can be checked mechanically, a judge for the parts that can't, and a small human-labelled set that keeps the judge honest. If you are still picking a framework, our Promptfoo vs Inspect comparison covers the tooling side.

The three judge formats

The 2023 MT-Bench paper by Zheng et al. (arXiv:2306.05685) described three ways to use a judge, and OpenAI's evaluation best practices list the same three today. Each answers a different question.

Scroll the table horizontally on smaller screens.

LLM judge formats and where they fit
FormatWhat the judge seesBest forTool support (examples)
Single-answer (pointwise)One output and a rubric; returns a score or pass/failRegression tests in CI, monitoring production samplesPromptfoo llm-rubric and g-eval; OpenAI score_model and label_model graders
PairwiseTwo outputs for the same input; picks the better oneComparing two prompts or models before a switchPromptfoo select-best (compares several outputs)
Reference-guidedThe output plus a gold answer or criterionFactual answers, RAG, questions with a known right answerInspect model_graded_qa and model_graded_fact
Formats from Zheng et al. (2023) and OpenAI evaluation best practices; tool features from Promptfoo, Inspect and OpenAI docs, accessed Nov 10, 2026.

Pointwise grading is the workhorse because it gives every output its own score you can track over time. Pairwise is better when you only need to know which of two versions wins, since "which is better" is an easier question than "how good is this on a scale of 1 to 10." OpenAI's guide recommends pairwise comparison or pass/fail "for more reliability." Reference-guided grading is the safest choice whenever a correct answer exists, because the judge compares against something concrete instead of its own opinion.

The biases you have to design around

Judges are not neutral. These are the failure modes with the strongest evidence, with the source for each.

Scroll the table horizontally on smaller screens.

Documented LLM judge biases
BiasWhat happensEvidenceMitigation
Position biasThe judge prefers an answer because of where it appearsWang et al. (2023): by changing answer order, Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT as the evaluatorRun pairwise checks in both orders; count a win only if it holds both ways
Verbosity biasLonger answers score higher even when they aren't betterNamed by Zheng et al. (2023); OpenAI's best-practices guide says LLMs bias towards longer responsesTell the judge length is not a criterion; add a length check in code; compare like-for-like lengths
Self-preferenceA model rates its own outputs higher than others' of equal qualityPanickssery et al. (2024) found GPT-4 and Llama 2 can recognise their own text and that self-recognition tracks self-preferenceUse a judge from a different model family than the system you are grading
Weak reasoning gradingJudges miss errors in math and multi-step logicZheng et al. (2023) list limited reasoning ability as a limitationGive a reference answer, or check these cases with code
Sources: Zheng et al., arXiv:2306.05685 (2023); Wang et al., arXiv:2305.17926 (2023); Panickssery et al., arXiv:2404.13076 (2024); OpenAI evaluation best practices, accessed Nov 10, 2026. Findings come from the models those papers tested and may differ for current models.

The self-preference point has a practical twist. In Inspect, a model-graded scorer with no grader model configured falls back to the model being evaluated, so it can end up grading itself. Inspect's model grading docs let you require a separate grader with ModelRole("grader", required=True), which raises an error instead of falling back. Promptfoo picks a default grader based on which API keys you have set, so pin the grader explicitly there too (llm-rubric docs).

Write a rubric the judge can actually follow

A common cause of bad judge scores is a vague rubric rather than a weak model. A rubric that says "is the answer helpful?" invites the judge to make up its own standard. These rules, drawn from Anthropic and OpenAI guidance, fix most of it.

  • Ask for pass/fail or a small scale. Anthropic recommends telling the judge to output only "correct" or "incorrect", or a score from 1 to 5. OpenAI's guide likewise recommends pass/fail or pairwise formats for more reliability.
  • One criterion per check. Grade tone, accuracy and completeness separately. Promptfoo's g-eval grades each criterion in an array independently and averages the scores, which also tells you which criterion failed.
  • Spell out what fails. Write concrete rules, for example: if the answer doesn't name the refund deadline, it fails.
  • Show examples of each grade. OpenAI suggests few-shot examples of great, fair and poor answers in the judge prompt.
  • Reason first, score last. Both Anthropic and OpenAI recommend letting the judge reason before it gives a score. G-Eval (Liu et al., 2023) built its method on chain-of-thought plus a form-filling format.
  • Return structured output. Ask for JSON with a reason and a score so you can parse results reliably and read why something failed.

Here is an illustrative Promptfoo config that applies these rules. The grader is pinned and set to temperature 0, two criteria are graded separately, and a code check handles length so the judge doesn't have to. Model names are examples; use the judge you have validated.

# promptfooconfig.yaml (illustrative)
defaultTest:
options:
provider:
id: anthropic:messages:claude-sonnet-5 # judge from a different family than the app model
config:
temperature: 0
tests:
- vars:
question: "Can I get a refund after 30 days?"
assert:
- type: llm-rubric
value: |
PASS only if the answer states the 30-day refund window and
says how to contact support. FAIL if it promises a refund
outside the policy. Length and friendliness are not criteria.
- type: llm-rubric
value: Does not invent policy details that are not in the context.
- type: javascript
value: output.length < 1200

Promptfoo's llm-rubric returns a JSON object with a reason, a score from 0 to 1 and a pass flag, and its docs describe pinning temperature on a custom judge as "the supported way to push grading toward reproducibility" (llm-rubric docs). For pairwise checks between prompt variants, Promptfoo's select-best picks the best of several outputs for the same test. If you prefer code-first evals, Inspect's model_graded_qa takes custom instructions, a grade pattern and a list of grader models.

Check your judge against human labels

A judge is itself a model you need to evaluate. OpenAI's guide puts it plainly: validate agreement against your human labels before optimizing for cost or latency, and scale up only once the judge consistently agrees with human annotations. A simple process:

  • Label a sample by hand. Take real outputs, including failures and edge cases, and have one or two people grade them with the same rubric the judge will use. A few dozen to a hundred examples is a reasonable start (a Metarix rule of thumb, not a published threshold).
  • Balance the set. OpenAI warns that if one label dominates, the judge learns to guess it. Make sure failures are well represented.
  • Measure agreement two ways. Raw agreement is easy to read but flatters you when most answers pass. Cohen's kappa corrects for agreement expected by chance; scikit-learn's cohen_kappa_score computes it in one line.
  • Read the disagreements. Look at the confusion matrix. A judge that passes outputs humans failed is lenient and will hide regressions; one that fails good outputs is strict and will create noise.
  • Fix the rubric, then re-test. Add the cases the judge got wrong as examples or as new rules, and measure again.

This illustrative snippet uses 12 made-up labels to show the calculation. We ran it on the toy data shown; the numbers describe this example only.

from sklearn.metrics import cohen_kappa_score, confusion_matrix

# Illustrative toy labels: 1 = pass, 0 = fail, for the same 12 outputs
human = [1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 0]
judge = [1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0]

agreement = sum(h == j for h, j in zip(human, judge)) / len(human)
print(f"raw agreement: {agreement:.2f}") # 0.83
print(f"Cohen's kappa: {cohen_kappa_score(human, judge):.2f}") # 0.64
print(confusion_matrix(human, judge, labels=[1, 0]))
# [[7 0]
# [2 3]] -> the judge passed 2 outputs that humans failed

Here 83% raw agreement looks good, but kappa is lower at 0.64, and the confusion matrix shows the judge's only mistakes were passing two answers people rejected. That lenient pattern is the one to fix first, because it lets regressions through. For context, Zheng et al. reported that GPT-4 as a judge reached over 80% agreement with human preferences, about the same rate at which humans agreed with each other. Your task and rubric decide what your number will be, so measure it rather than assume it.

Keep the judge stable over time

A judge you validated in March can quietly change by June. Treat it like any other dependency.

  • Pin the judge model version. Promptfoo chooses a default grader from whichever API keys are present, so a new key on a CI runner can swap your judge. OpenAI's score_model grader only accepts listed dated snapshots, which helps here.
  • Version the rubric with the code. A rubric edit changes every score, so keep it in the repo and note it next to results.
  • Re-run the human check when anything changes. New judge model, new rubric or a new kind of input all mean re-measuring agreement.
  • Consider more than one judge for important calls. Inspect can pass a list of grader models that each grade independently, with the grades combined by a reducer.
  • Watch the cost. Each judged test is another model call on top of the one being graded. Use a strong judge while you calibrate, then try a cheaper one and keep it only if agreement holds.

Finally, remember what a judge score is: an estimate of how humans would grade, on your data. It says nothing about how a model ranks on public leaderboards, which use their own judges and setups; our guide to reading LLM leaderboards explains why those numbers rarely transfer to your product. Some public benchmarks use judges too: HELMET, for example, grades long-document QA and summarization with GPT-4o, as our long-context benchmarks explainer describes.

A quick checklist

  • Could a code check grade this instead? Use code where you can.
  • Pointwise for tracking, pairwise for choosing, reference-guided when a right answer exists.
  • Pass/fail or a small scale, one criterion per check, concrete fail rules.
  • Judge from a different model family, pinned version, temperature 0 where supported.
  • Pairwise checks run in both orders; length handled separately.
  • Agreement and kappa measured against human labels before you rely on scores.
  • Re-validate whenever the judge, rubric or input mix changes.

Verdict

LLM-as-a-judge is worth using for the open-ended parts of your evals, the parts code can't check and humans can't review at scale. But a judge is only as good as its rubric and its agreement with people. Keep grades binary or on a short scale, split criteria, use a judge from a different model family, and control for answer order and length.

Most importantly, don't trust a judge you haven't measured. Label a balanced sample by hand, check raw agreement and kappa, read the disagreements, and repeat whenever anything changes. Promptfoo is the quicker way to add rubric checks to CI; Inspect gives more control for agent evals and multi-judge setups. Either way, the human-labelled set is what makes the scores mean something.

Sources

Tools covered in this guide

Frequently asked questions

What is LLM-as-a-judge?
It is the practice of using one language model to grade another model's output against a rubric, a reference answer or a competing output. It is used for open-ended qualities such as accuracy, completeness or tone that exact-match code checks can't measure.
Is an LLM judge as reliable as human reviewers?
It can come close on some tasks. Zheng et al. (2023) found GPT-4 as a judge agreed with human preferences over 80% of the time, about the same rate humans agreed with each other. But reliability depends on your task and rubric, so measure agreement against your own human labels before relying on it.
Should I use pairwise or single-answer grading?
Use single-answer (pointwise) grading to track quality over time and in CI. Use pairwise comparison when you need to choose between two prompts or models, and run it in both orders to cancel out position bias. If a correct answer exists, reference-guided grading is usually the most reliable.
Can a model grade its own outputs?
It can, but it shouldn't for evaluations that matter. Research by Panickssery et al. (2024) found LLM evaluators recognise and favour their own generations. Use a judge from a different model family, and in tools such as Inspect require an explicit grader model so it never falls back to the model being tested.

More in Quality, cost & reliability

Share