How do you measure whether your AI agent is getting better each sprint?

Run the same fixed test set on every sprint build. Track the pass rate for each failure type, not one blended score. Show a confidence interval so noise is not read as progress. Check consistency across repeated runs. Keep a regression suite so a fix this sprint does not break last sprint’s win.

Why is “it feels better” not enough?

Agent output changes from run to run, and small test sets are noisy. A score that rises by two points may be luck. The tau-bench paper shows how large the inconsistency can be. It reports that even state-of-the-art function-calling agents are inconsistent across repeated trials, and that pass^8, its new reliability metric, was below 25% in the retail domain (Yao et al., arXiv 2406.12045, 2024-06).

So a good sprint measure has three parts: a fixed set of tasks, scores that people trust, and a way to separate real change from noise.

What do you need before the first sprint measurement?

1. A test set from real failures. Anthropic’s engineering team writes that “20-50 simple tasks drawn from real failures is a great start”. Early changes tend to have large effects, so small sets can still show them. Waiting for a large suite makes evals harder to build (Anthropic, 2026-01-09).
2. A failure list. Review real traces and count how often each failure happens. The evals FAQ recommends reviewing at least 100 traces (Husain and Shankar, Evals FAQ, 2026-09-18). Hamel Husain calls error analysis “the single most valuable activity in AI development” (Hamel Husain, 2025-03-24).
3. Graders you trust. Combine code checks with LLM judges where needed. Anthropic says LLM-as-judge graders “should be closely calibrated with human experts” (Anthropic, 2026-01-09). See how many labeled examples you need to trust an LLM judge.
4. A frozen baseline. Run the current agent once and record the results. Every later sprint compares to it.

Which metrics should you track each sprint?

Use a small set. One table per sprint is enough.

Metric	What it tells you	Source of the idea
Pass rate per failure type	Whether the fix you built this sprint worked	Husain: track pass rates on specific failure categories
Overall pass rate with 95% interval	Whether the net change is larger than noise	Miller: report standard error with each score
Regression suite pass rate	Whether old wins still hold	Anthropic: regression evals should pass nearly every time
pass^k over repeated runs	Whether the agent is consistent, not just sometimes right	Yao et al., Anthropic
Judge agreement with experts	Whether the scores can be trusted at all	Husain, Yan, Arize

Husain gives a worked case. At one company, date-handling success went from 33% to 95% after the team targeted that failure. He also notes that “a 10% increase in passing outputs is immediately meaningful” (Hamel Husain, 2025-03-24). Those are his reported figures for a different company, not a promise for your agent.

The evals FAQ gives the habit to copy. After a fix, check whether that specific problem’s rate dropped, and keep a running log of the errors you caught, the fix and the impact (Husain and Shankar, Evals FAQ, 2026-09-18).

How do you tell progress from noise?

Evan Miller’s paper treats evals as experiments. His recommendations are practical:

• Report the standard error of the mean next to every score, and compute a 95% confidence interval from it.
• When two versions run on the same questions, analyze the per-question differences. He recommends “the paired version of the standard error estimate wherever practicable.”
• Use clustered standard errors when questions come in related groups.
• Reduce noise by sampling answers repeatedly.
• Use power analysis to check whether the eval can detect the difference you care about.

(Miller, arXiv 2411.00640, 2024-11)

In practice, run the old and new agent on the same tasks. Compare task by task. If the interval for the difference includes zero, report “no clear change” and do not call it a win.

Why keep two kinds of evals?

Anthropic separates them. Capability evals ask what the agent can do, and should start with low pass rates to leave room to improve. Regression evals ask “Does the agent still handle all the tasks it used to?” and should pass nearly every time. When a capability eval passes reliably, it can move into the regression suite (Anthropic, 2026-01-09).

This split gives a clean sprint routine:

1. Run the regression suite first. A drop blocks the release.
2. Run the capability evals for this sprint’s target failures.
3. Move tasks that now pass reliably into the regression suite.
4. Add new failures found in production to the capability set.

Anthropic also warns about saturation. When an agent passes all the solvable tasks, the eval stops showing progress, and you need harder tasks.

What mistakes make sprint scores misleading?

• Changing the test set between sprints. Scores are no longer comparable. Add tasks, but keep a frozen core.
• Trusting an unchecked judge. If the judge drifts from expert opinion, the trend line is meaningless.
• Reading only the score. Anthropic’s advice is “Read the transcripts!” Transcripts show whether a failure is a real mistake or a grader rejecting a valid answer.
• Using pass@k for a customer-facing agent. Anthropic notes pass@k rises with more attempts, while pass^k asks whether every trial succeeds. Pass^k matters more where users expect consistency.
• Using 1 to 5 scales without definitions. The evals FAQ says scale scores need larger samples to detect real differences. It favors binary checks.

FAQ

How big should the test set be?

Start small. Anthropic suggests 20 to 50 tasks drawn from real failures. Grow it as new failures appear. For more reliable comparisons, use power analysis to check whether the set can detect the change you care about.

How often should we re-run the full evaluation?

Run the regression suite on every build that changes the agent. Run the full capability set at least once per sprint. The sources do not set a fixed schedule, so choose one that matches your release rhythm and cost.

What if the score goes up but users are not happier?

The test set may not match real use. Anthropic’s documentation says evals should “mirror your real-world task distribution” (Anthropic docs, accessed 2026-10-30). Add failures found in production and review transcripts.

Do we need an eval tool or a service?

Either can run the loop. Tools run the tests. People decide what to test and what good looks like. See eval tool vs eval service.

How Clarity can help

Clarity by Epistemic Me is an AI-native services firm. Clarity helps teams set up the measurement loop in this guide: error analysis, expert rubrics, golden datasets, LLM-as-judge checked against expert labels, and regression gates. Engagements start at $100K+. Sprint Zero, the first phase, reviews inputs, establishes a baseline and agrees the first increment. Lynn Chyi, AI Experience Architect at Dayforce, said the work improved “our human expert alignment scores by nearly 60%” (case study). See /evals/, /sprint-zero/ and how to evaluate an AI agent.

Sources

• Yao et al. — tau-bench (2024-06)
• Evan Miller — Adding Error Bars to Evals (2024-11)
• Anthropic — Demystifying evals for AI agents (2026-01-09)
• Anthropic — Define success criteria and build evaluations (accessed 2026-10-30)
• Hamel Husain — A Field Guide to Rapidly Improving AI Products (2025-03-24)
• Hamel Husain and Shreya Shankar — Frequently Asked Questions about AI Evals (2026-09-18)
• Eugene Yan — Evaluating the Effectiveness of LLM-Evaluators (2024-08)
• Arize — How to measure human-LLM judge alignment (2026-07)