How many labeled examples do you need before you trust an LLM judge?

There is no single number. Practitioners suggest about 100 expert-labeled examples per failure mode, with enough Pass and Fail cases to measure both. Below 60 examples, confidence intervals are often too wide for a useful conclusion. Trust comes from the interval around the judge’s true positive and negative rates on a held-out test set, not from label count.

Why does the number of labels matter at all?

An LLM judge is a model that grades the output of another model. It is only useful if its grades match what your domain expert would say. An influential 2023 paper on LLM-as-a-judge reports that strong judges such as GPT-4 reach “over 80% agreement, the same level of agreement between humans” (Zheng et al., arXiv 2306.05685, 2023-06).

That result comes from general chat benchmarks. It does not tell you how a judge will behave on your task, your rubric and your failure modes. Another group of researchers states the reason plainly: LLM-generated evaluators inherit the problems of the models they judge, so they still need human validation (Shankar et al., arXiv 2404.12272, 2024-04).

Expert labels are how you do that validation. The question is how many you need to be confident in the result.

What do practitioners say about sample size?

Sources give different numbers. They also measure different things. The table lists what each one says.

Source	What it says about count	What it counts
Hamel Husain, LLM judge guide (2024-10-29)	About 100 examples per failure mode. Below 60 examples, confidence intervals are often too wide for a useful conclusion.	Labeled examples used to validate a judge
Hamel Husain, evals-skills validate-evaluator (accessed 2026-10-30)	About 100 labeled traces, balanced at roughly 50 Pass and 50 Fail. Also 30 to 50 examples per class across dev and test combined.	Labeled traces split into train, dev and test
Eugene Yan, LLM evaluators (2024-08)	No minimum is given for validating an evaluator. He notes that during development you will likely evaluate only a few hundred samples.	Samples evaluated during development
Arize, measuring human-LLM judge alignment (2026-07)	About 100 examples gives a margin near plus or minus 10 points on raw agreement. About 400 gives near plus or minus 5 points. These are rough worst-case figures for intuition only.	Examples used to estimate raw agreement

The sources do not agree on one number. They do agree on three points:

1. Roughly 100 labels is a common starting point, not a guarantee.
2. Below about 60 labels, confidence intervals are often too wide for a useful conclusion.
3. The count that matters most is the count of the rare class. For many agent tasks that class is the failures.

Why is the number of failures the real limit?

Most agent outputs pass. Failures are rare. A judge that says “Pass” every time can still show high raw agreement with humans, yet it catches no errors. Arize gives the arithmetic: a 5% failure rate in 200 examples leaves only about ten failures, which makes recall unstable (Arize, 2026-07).

So count failures, not total labels. This is why the evals-skills guide asks for a balanced set of about 50 Pass and 50 Fail, even if the real traffic is skewed (Hamel Husain, evals-skills, accessed 2026-10-30). Eugene Yan reports the same effect in the studies he reviews: one set of evaluators identified over 95% of consistent summaries but only 30 to 60% of inconsistent ones (Eugene Yan, 2024-08).

If you cannot find enough failures in your logs, you cannot yet trust a judge on that failure mode. Find more failures first.

Which metrics should you report?

Use more than one number.

• True positive rate (TPR): the share of expert Pass labels that the judge also passes.
• True negative rate (TNR): the share of expert Fail labels that the judge also fails.
• Raw agreement: easy to read, but it ignores chance and class imbalance.
• Cohen’s kappa: agreement corrected for chance. Sources differ on its use: Arize and Eugene Yan use it for judges, while the evals-skills guide keeps it for human-human agreement only. Eugene Yan notes it is “generally more conservative” than other metrics. In one study he cites, a model had 80% agreement but only 0.62 kappa (Eugene Yan, 2024-08).
• Human-human baseline: compare the judge against how often two experts agree with each other. Arize adds a caution: human agreement is “a baseline, not a hard performance ceiling” (Arize, 2026-07).

Arize also warns against universal thresholds. The evals-skills guide does give targets: TPR and TNR above 90% on the dev set, and above 80% as a minimum (Hamel Husain, evals-skills). Treat these as one practitioner’s starting rules, not a standard. Set your own bar from the cost of a wrong grade in your product.

How do you put a confidence interval on a judge?

A point estimate such as “92% TNR” means little without its range. Two sources describe practical methods.

• The evals-skills guide uses a bootstrap: 2,000 resamples of the test set, then the 2.5th and 97.5th percentiles as a 95% interval.
• Arize suggests bootstrapping over examples for kappa and F1, resampling whole traces when parts of a trace are correlated, and using a paired bootstrap when you compare two judges.

For the underlying statistics of eval scores, Evan Miller’s paper recommends reporting the standard error next to each score and computing a 95% confidence interval from it (Miller, arXiv 2411.00640, 2024-11).

What is a sound process, step by step?

1. Pick one domain expert whose judgment defines quality. The evals FAQ advises a single “benevolent dictator” for most teams (Husain and Shankar, Evals FAQ, 2026-09-18).
2. Review traces and find failure modes. Start with roughly 30 examples and continue until new failure modes stop appearing (Hamel Husain, 2024-10-29).
3. Collect about 100 labels per failure mode, balanced Pass and Fail where you can.
4. Split the data. The guides suggest about 10 to 20% for few-shot examples, 40 to 45% for dev and 40 to 45% for test (Husain and Shankar, Evals FAQ, 2026-09-18).
5. Tune the judge prompt on dev only. Check the test set once.
6. Report TPR, TNR and their 95% intervals. If test scores fall well below dev scores, the judge may be overfitting.
7. Re-run the check after material changes, such as a model update (Hamel Husain, 2024-10-29).

Anthropic’s engineering team gives the same direction for agent evals: LLM-as-judge graders “should be closely calibrated with human experts” (Anthropic, 2026-01-09).

FAQ

Is 50 labeled examples enough to trust an LLM judge?

Usually not. Hamel Husain says that below 60 examples, confidence intervals are often too wide for a useful conclusion. Fifty can be enough to find failure modes and write a first rubric. It is not enough to validate a judge.

Can I use raw agreement as my only metric?

No. Raw agreement hides a judge that always says Pass when failures are rare. Report TPR and TNR separately. Sources differ on kappa: Arize and Eugene Yan use it for judges, while the evals-skills guide keeps it for human-human agreement only. Show the confusion matrix too.

What if my experts disagree with each other?

Measure it. Compare expert-expert agreement with judge-expert agreement on the same examples. If the experts disagree often, the rubric is unclear. Ask which rubric part caused the split, write a rule that settles it, and relabel the affected examples (Husain and Shankar, Evals FAQ, 2026-09-18).

Does the sample size change if I use a stronger judge model?

The sources do not say that a stronger model needs fewer labels. A judge is validated against your task and your rubric. Re-run the check when you change the judge model.

How Clarity can help

Clarity by Epistemic Me is an AI-native services firm. Clarity’s evals consulting covers error analysis, expert rubrics, golden datasets, and LLM-as-judge checked against expert labels, with regression gates. Engagements start at $100K+, with scope, deliverables and duration agreed in the proposal. Lynn Chyi, AI Experience Architect at Dayforce, said the work improved “our human expert alignment scores by nearly 60%” (case study). Clarity is an official partner of Parlance Labs. See /evals/ and the related answer, how to evaluate an AI agent. For the judge itself, read Clarity’s LLM-as-judge guide.

Sources

• Zheng et al. — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023-06)
• Shankar et al. — Who Validates the Validators? (2024-04)
• Evan Miller — Adding Error Bars to Evals (2024-11)
• Hamel Husain — Creating a LLM-as-a-Judge (2024-10-29)
• Hamel Husain — evals-skills, validate-evaluator (accessed 2026-10-30)
• Hamel Husain and Shreya Shankar — Frequently Asked Questions about AI Evals (2026-09-18)
• Eugene Yan — Evaluating the Effectiveness of LLM-Evaluators (2024-08)
• Arize — How to measure human-LLM judge alignment (2026-07)
• Anthropic — Demystifying evals for AI agents (2026-01-09)