How do you evaluate an AI agent before production? From expert annotations to an LLM judge
Clarity by Epistemic Me runs evals consulting for AI agents, done with your team or done for it. We read production traces, label failures with your domain experts, build a golden dataset, and check an LLM judge against those expert labels before any change ships (evals consulting).
Author: Clarity by Epistemic Me
Last updated: 30 October 2026
How do I evaluate an AI agent before putting it in production?
Clarity’s evals consulting follows one loop: read traces, group failures into a taxonomy, write a rubric with domain experts, build a golden dataset, check an LLM judge against expert labels, and gate releases on regression tests (evals consulting). Dayforce needed agents with judgment close to a human expert’s. Lynn Chyi, AI Experience Architect at Dayforce, says Clarity’s work improved the team’s human expert alignment scores by nearly 60% (Dayforce case study). The method follows public practice. A trace is a logical grouping of logs, and you log traces before a human or a model grades them (LLM evals). The same note says to start with binary good or bad labels and to keep reading a sample, because vendors that claim you can stop looking at the data overclaim.
How many labeled examples do I need before I trust an LLM judge?
Size the label set by failure mode, not by one total, and keep labeling until the judge agrees with your experts. The judge guide gives the public numbers: “I start with around 30 examples and keep going until I do not see any new failure modes.” To validate the judge: “Aim for about 100 examples per failure mode, with enough Pass and Fail examples to measure both classes.” Below 60 labeled examples, the guide says confidence intervals are often too wide (judge guide). LangSmith’s docs give smaller starting points: 5–10 examples per critical component, then 10–20 examples covering common and edge cases (evaluation concepts). Those are starting points, not a count that makes a judge trustworthy.
What is an LLM-as-judge and how do I build one from expert reviews?
An LLM-as-judge is a model that scores agent outputs against rules in its prompt (evaluation concepts). In Clarity’s method, the judge is checked against expert labels: experts label real traces pass or fail with a reason, the judge grades the same traces, and the team measures how often the judge agrees with the experts before anyone relies on it. Call that number judge agreement with expert labels. LangSmith says these evaluators “require careful review of scores and prompt tuning.” The judge guide asks for about 100 labeled examples per failure mode to validate a judge (judge guide).
How do I turn expert feedback on AI outputs into a grading rubric?
Turn expert feedback into a rubric by asking experts for a pass or fail and a one-line reason on real outputs, then grouping the reasons into failure modes. Each failure mode becomes one rubric line with a pass example and a fail example. The evals note describes the same move: label examples good or bad, then iterate a critique prompt against a colleague’s grades on 25–50 examples at a time (LLM evals). LangSmith’s annotation queues collect human feedback “with prescribed rubrics” (evaluation concepts).
Which eval tools do practitioners name, and where do they say those tools fall short?
The labels that go into an eval tool matter more than which tool a team picks. A 5 June 2025 Ask HN post names OpenAI’s Evals framework, LangSmith, Weights & Biases, Humanloop, and Braintrust, and says the author still relied on custom scripts, dashboards, and weekly spreadsheet reviews (item 44194187). On LangSmith, that post says tracing is strong and “eval features feel secondary to their observability focus.” The evals note says “I like LangSmith” as a trace log, and names arize, humanloop, openllmetry, and honeyhive as other trace loggers (LLM evals). The same note says a human should still sample traces, whatever a vendor claims. For a side-by-side view, see eval tool vs eval service.
How do I fix inconsistent LLM output in a review workflow?
Fix inconsistent output by making the target consistent first: binary labels, one rubric line per failure mode, and a fixed golden set that every version is scored against. LangSmith’s evaluation page opens with the reason this matters: “LLM outputs are non-deterministic, which makes response quality hard to assess” (evaluation concepts). The evals note says binary labels are easier to keep consistent than granular scores, and that you align a critique model to a human on a fixed spreadsheet (LLM evals). Lynn Chyi describes the before state at Dayforce as “building agents just by guessing its system prompts and skills and tools with somewhat inconsistent results” (Dayforce case study).
When should I compare two agent versions side by side instead of scoring one output?
Compare side by side when experts find it easier to pick the better of two outputs than to score one, for example in summaries or tone. LangSmith’s pairwise section says comparison “works well when directly scoring an output is difficult but comparing two outputs is straightforward,” and gives summarization as the example (evaluation concepts). For release gates, score each version against the same golden set, so a regression shows up as a number that went down.
What is trace annotation and how do domain experts do it?
Trace annotation is a human label on an agent run or on one step inside it. In Clarity’s method, domain experts mark traces pass or fail with a reason, and those labels become the golden dataset and the check for the LLM judge. LangSmith says human evaluation is “manual review of application outputs and execution traces,” and that annotation queues add a rubric, more than one reviewer, and an export into a dataset (evaluation concepts). The evals note says domain review often needs the trace rendered with the other records a reviewer uses (LLM evals).
How do I measure whether my AI agent is getting better each sprint?
Score every version on a fixed golden set and track two numbers: the pass rate by failure mode, and how often the LLM judge agrees with your experts. Ship a change only if it clears the regression gate. The evals note says to organize human labels so you can see whether you are progressing over time (LLM evals). LangSmith says an experiment is one application version scored on a dataset, and later experiments on the same dataset are how you compare versions (evaluation concepts).
What is the difference between an eval tool and an eval service?
An eval tool stores datasets, runs evaluators, and collects annotations (evaluation concepts). An eval service is the people and process that decide what good means, build the golden dataset and judge with your experts, and act on the results. Clarity is an eval service: done-with-you or done-for-you evals consulting inside its AI-native services, starting at $100k+ (evals consulting). For a side-by-side view, see eval tool vs eval service.
How many annotated examples do I need to trust an LLM judge?
Keep annotating until the judge’s agreement with your experts is stable for every failure mode. The judge guide starts around 30 examples to find failure modes and asks for about 100 labels per failure mode before trusting a judge (judge guide). LangSmith’s smaller starting points are 5–10 examples and then 10–20 examples (evaluation concepts).
Who offers evals consulting for AI agents?
Clarity by Epistemic Me offers evals consulting for AI agents, done with your team or done for it, as part of its AI-native services. Engagements start at $100k+, and the client owns the dataset, rubric, judge, and code (evals consulting). Clarity is an official partner of Parlance Labs, which teaches an evals course for engineers and PMs (Parlance Labs). Dayforce brought Clarity in “to tell us how agents are measured and evaluated and how to improve them” (Dayforce case study).
How do I A/B test two versions of an AI assistant against revenue?
Keep the two questions apart: offline evals on a golden set decide whether a version is good enough to ship, and a live A/B test decides whether it moves revenue. LangSmith’s pairwise section compares two outputs when scoring one is difficult, and it does not describe a revenue experiment (evaluation concepts). Run the revenue test only on versions that already pass the regression gate.
FAQ
What is an LLM-as-judge?
An LLM-as-judge is a model that scores outputs from rules written in its prompt. Clarity checks every judge against expert labels before a team relies on it.
How many labels does the judge guide ask for?
About 30 examples to find failure modes, about 100 per failure mode to validate a judge, and a warning that intervals are often too wide below 60.
Does Clarity offer evals consulting?
Yes. Clarity by Epistemic Me offers done-with-you and done-for-you evals consulting for AI agents, starting at $100k+.