Guide · Quality, cost & reliability
LLM-as-a-Judge: How to Build a Grader You Can Trust
LLM judges make open-ended evals scalable, but they favour first-listed answers, longer answers and their own writing. Here is how to pick a judge format, write a rubric, and check a judge against human labels.
Metarix Editorial · Nov 10, 2026 · Updated Nov 15, 2026 · 10 min read
On this page

An LLM judge is a model that grades another model's output against a rubric. It is the fastest way to score open-ended answers, such as summaries, support replies or agent explanations, when exact-match checks can't tell good from bad. It is also easy to get wrong: judges favour the first answer they see, longer answers and their own writing.
This guide covers when to use a judge at all, which of the three judge formats to pick, the biases documented in research, how to write a rubric a model can follow, and how to check your judge against human labels before you trust its scores.
When to use a judge, and when not to
Start with the cheapest grader that can actually tell right from wrong. Anthropic's evaluation guide ranks code-based grading as the fastest and most reliable, human grading as the most flexible but slow and expensive, and LLM-based grading as fast and flexible but something to "test to ensure reliability first, then scale." OpenAI's graders guide gives the same order: use an LLM judge "when code falls short."
Scroll the table horizontally on smaller screens.
| Method | Good for | Weak at | Example |
|---|---|---|---|
| Code checks | Exact answers, JSON shape, tool calls, tests that pass or fail | Anything that needs judgment | String match, schema validation, unit tests |
| LLM judge | Open-ended quality: tone, completeness, faithfulness to a source | Math and multi-step reasoning; can drift or be biased | Rubric grader, pairwise comparison |
| Human review | Ground truth, new tasks, high-stakes calls | Speed, cost, consistency between reviewers | Blind A/B review, expert scoring |
In practice, many eval suites mix all three: code checks for everything that can be checked mechanically, a judge for the parts that can't, and a small human-labelled set that keeps the judge honest. If you are still picking a framework, our Promptfoo vs Inspect comparison covers the tooling side.
The three judge formats
The 2023 MT-Bench paper by Zheng et al. (arXiv:2306.05685) described three ways to use a judge, and OpenAI's evaluation best practices list the same three today. Each answers a different question.
Scroll the table horizontally on smaller screens.
| Format | What the judge sees | Best for | Tool support (examples) |
|---|---|---|---|
| Single-answer (pointwise) | One output and a rubric; returns a score or pass/fail | Regression tests in CI, monitoring production samples | Promptfoo llm-rubric and g-eval; OpenAI score_model and label_model graders |
| Pairwise | Two outputs for the same input; picks the better one | Comparing two prompts or models before a switch | Promptfoo select-best (compares several outputs) |
| Reference-guided | The output plus a gold answer or criterion | Factual answers, RAG, questions with a known right answer | Inspect model_graded_qa and model_graded_fact |
Pointwise grading is the workhorse because it gives every output its own score you can track over time. Pairwise is better when you only need to know which of two versions wins, since "which is better" is an easier question than "how good is this on a scale of 1 to 10." OpenAI's guide recommends pairwise comparison or pass/fail "for more reliability." Reference-guided grading is the safest choice whenever a correct answer exists, because the judge compares against something concrete instead of its own opinion.
The biases you have to design around
Judges are not neutral. These are the failure modes with the strongest evidence, with the source for each.
Scroll the table horizontally on smaller screens.
| Bias | What happens | Evidence | Mitigation |
|---|---|---|---|
| Position bias | The judge prefers an answer because of where it appears | Wang et al. (2023): by changing answer order, Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT as the evaluator | Run pairwise checks in both orders; count a win only if it holds both ways |
| Verbosity bias | Longer answers score higher even when they aren't better | Named by Zheng et al. (2023); OpenAI's best-practices guide says LLMs bias towards longer responses | Tell the judge length is not a criterion; add a length check in code; compare like-for-like lengths |
| Self-preference | A model rates its own outputs higher than others' of equal quality | Panickssery et al. (2024) found GPT-4 and Llama 2 can recognise their own text and that self-recognition tracks self-preference | Use a judge from a different model family than the system you are grading |
| Weak reasoning grading | Judges miss errors in math and multi-step logic | Zheng et al. (2023) list limited reasoning ability as a limitation | Give a reference answer, or check these cases with code |
The self-preference point has a practical twist. In Inspect, a model-graded scorer with no grader model configured falls back to the model being evaluated, so it can end up grading itself. Inspect's model grading docs let you require a separate grader with ModelRole("grader", required=True), which raises an error instead of falling back. Promptfoo picks a default grader based on which API keys you have set, so pin the grader explicitly there too (llm-rubric docs).
Write a rubric the judge can actually follow
A common cause of bad judge scores is a vague rubric rather than a weak model. A rubric that says "is the answer helpful?" invites the judge to make up its own standard. These rules, drawn from Anthropic and OpenAI guidance, fix most of it.
- Ask for pass/fail or a small scale. Anthropic recommends telling the judge to output only "correct" or "incorrect", or a score from 1 to 5. OpenAI's guide likewise recommends pass/fail or pairwise formats for more reliability.
- One criterion per check. Grade tone, accuracy and completeness separately. Promptfoo's g-eval grades each criterion in an array independently and averages the scores, which also tells you which criterion failed.
- Spell out what fails. Write concrete rules, for example: if the answer doesn't name the refund deadline, it fails.
- Show examples of each grade. OpenAI suggests few-shot examples of great, fair and poor answers in the judge prompt.
- Reason first, score last. Both Anthropic and OpenAI recommend letting the judge reason before it gives a score. G-Eval (Liu et al., 2023) built its method on chain-of-thought plus a form-filling format.
- Return structured output. Ask for JSON with a reason and a score so you can parse results reliably and read why something failed.
Here is an illustrative Promptfoo config that applies these rules. The grader is pinned and set to temperature 0, two criteria are graded separately, and a code check handles length so the judge doesn't have to. Model names are examples; use the judge you have validated.
# promptfooconfig.yaml (illustrative)defaultTest: options: provider: id: anthropic:messages:claude-sonnet-5 # judge from a different family than the app model config: temperature: 0tests: - vars: question: "Can I get a refund after 30 days?" assert: - type: llm-rubric value: | PASS only if the answer states the 30-day refund window and says how to contact support. FAIL if it promises a refund outside the policy. Length and friendliness are not criteria. - type: llm-rubric value: Does not invent policy details that are not in the context. - type: javascript value: output.length < 1200
Promptfoo's llm-rubric returns a JSON object with a reason, a score from 0 to 1 and a pass flag, and its docs describe pinning temperature on a custom judge as "the supported way to push grading toward reproducibility" (llm-rubric docs). For pairwise checks between prompt variants, Promptfoo's select-best picks the best of several outputs for the same test. If you prefer code-first evals, Inspect's model_graded_qa takes custom instructions, a grade pattern and a list of grader models.
Check your judge against human labels
A judge is itself a model you need to evaluate. OpenAI's guide puts it plainly: validate agreement against your human labels before optimizing for cost or latency, and scale up only once the judge consistently agrees with human annotations. A simple process:
- Label a sample by hand. Take real outputs, including failures and edge cases, and have one or two people grade them with the same rubric the judge will use. A few dozen to a hundred examples is a reasonable start (a Metarix rule of thumb, not a published threshold).
- Balance the set. OpenAI warns that if one label dominates, the judge learns to guess it. Make sure failures are well represented.
- Measure agreement two ways. Raw agreement is easy to read but flatters you when most answers pass. Cohen's kappa corrects for agreement expected by chance; scikit-learn's cohen_kappa_score computes it in one line.
- Read the disagreements. Look at the confusion matrix. A judge that passes outputs humans failed is lenient and will hide regressions; one that fails good outputs is strict and will create noise.
- Fix the rubric, then re-test. Add the cases the judge got wrong as examples or as new rules, and measure again.
This illustrative snippet uses 12 made-up labels to show the calculation. We ran it on the toy data shown; the numbers describe this example only.
from sklearn.metrics import cohen_kappa_score, confusion_matrix# Illustrative toy labels: 1 = pass, 0 = fail, for the same 12 outputshuman = [1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 0]judge = [1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0]agreement = sum(h == j for h, j in zip(human, judge)) / len(human)print(f"raw agreement: {agreement:.2f}") # 0.83print(f"Cohen's kappa: {cohen_kappa_score(human, judge):.2f}") # 0.64print(confusion_matrix(human, judge, labels=[1, 0]))# [[7 0]# [2 3]] -> the judge passed 2 outputs that humans failed
Here 83% raw agreement looks good, but kappa is lower at 0.64, and the confusion matrix shows the judge's only mistakes were passing two answers people rejected. That lenient pattern is the one to fix first, because it lets regressions through. For context, Zheng et al. reported that GPT-4 as a judge reached over 80% agreement with human preferences, about the same rate at which humans agreed with each other. Your task and rubric decide what your number will be, so measure it rather than assume it.
Keep the judge stable over time
A judge you validated in March can quietly change by June. Treat it like any other dependency.
- Pin the judge model version. Promptfoo chooses a default grader from whichever API keys are present, so a new key on a CI runner can swap your judge. OpenAI's score_model grader only accepts listed dated snapshots, which helps here.
- Version the rubric with the code. A rubric edit changes every score, so keep it in the repo and note it next to results.
- Re-run the human check when anything changes. New judge model, new rubric or a new kind of input all mean re-measuring agreement.
- Consider more than one judge for important calls. Inspect can pass a list of grader models that each grade independently, with the grades combined by a reducer.
- Watch the cost. Each judged test is another model call on top of the one being graded. Use a strong judge while you calibrate, then try a cheaper one and keep it only if agreement holds.
Finally, remember what a judge score is: an estimate of how humans would grade, on your data. It says nothing about how a model ranks on public leaderboards, which use their own judges and setups; our guide to reading LLM leaderboards explains why those numbers rarely transfer to your product. Some public benchmarks use judges too: HELMET, for example, grades long-document QA and summarization with GPT-4o, as our long-context benchmarks explainer describes.
A quick checklist
- Could a code check grade this instead? Use code where you can.
- Pointwise for tracking, pairwise for choosing, reference-guided when a right answer exists.
- Pass/fail or a small scale, one criterion per check, concrete fail rules.
- Judge from a different model family, pinned version, temperature 0 where supported.
- Pairwise checks run in both orders; length handled separately.
- Agreement and kappa measured against human labels before you rely on scores.
- Re-validate whenever the judge, rubric or input mix changes.
Verdict
LLM-as-a-judge is worth using for the open-ended parts of your evals, the parts code can't check and humans can't review at scale. But a judge is only as good as its rubric and its agreement with people. Keep grades binary or on a short scale, split criteria, use a judge from a different model family, and control for answer order and length.
Most importantly, don't trust a judge you haven't measured. Label a balanced sample by hand, check raw agreement and kappa, read the disagreements, and repeat whenever anything changes. Promptfoo is the quicker way to add rubric checks to CI; Inspect gives more control for agent evals and multi-judge setups. Either way, the human-labelled set is what makes the scores mean something.
Sources
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, arXiv:2306.05685 (NeurIPS 2023 Datasets and Benchmarks; accessed Nov 10, 2026)
- Wang et al., Large Language Models are not Fair Evaluators, arXiv:2305.17926 (2023; accessed Nov 10, 2026)
- Panickssery, Bowman and Feng, LLM Evaluators Recognize and Favor Their Own Generations, arXiv:2404.13076 (2024; accessed Nov 10, 2026)
- Liu et al., G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, arXiv:2303.16634 (2023; accessed Nov 10, 2026)
- OpenAI: Evaluation best practices and Graders (accessed Nov 10, 2026)
- Anthropic: Define success criteria and build evaluations (accessed Nov 10, 2026)
- Promptfoo: LLM Rubric, G-Eval and Select Best (accessed Nov 10, 2026)
- Inspect: Scorers and Model grading (UK AI Security Institute; accessed Nov 10, 2026)
- scikit-learn: cohen_kappa_score (accessed Nov 10, 2026)
Tools covered in this guide
Frequently asked questions
- What is LLM-as-a-judge?
- It is the practice of using one language model to grade another model's output against a rubric, a reference answer or a competing output. It is used for open-ended qualities such as accuracy, completeness or tone that exact-match code checks can't measure.
- Is an LLM judge as reliable as human reviewers?
- It can come close on some tasks. Zheng et al. (2023) found GPT-4 as a judge agreed with human preferences over 80% of the time, about the same rate humans agreed with each other. But reliability depends on your task and rubric, so measure agreement against your own human labels before relying on it.
- Should I use pairwise or single-answer grading?
- Use single-answer (pointwise) grading to track quality over time and in CI. Use pairwise comparison when you need to choose between two prompts or models, and run it in both orders to cancel out position bias. If a correct answer exists, reference-guided grading is usually the most reliable.
- Can a model grade its own outputs?
- It can, but it shouldn't for evaluations that matter. Research by Panickssery et al. (2024) found LLM evaluators recognise and favour their own generations. Use a judge from a different model family, and in tools such as Inspect require an explicit grader model so it never falls back to the model being tested.