What should an AI agent pilot deliver in the first month?

An AI agent pilot should deliver five things in its first month: one scoped workflow, a measured baseline of how the work is done today, written pass/fail criteria, a small test set built from real cases, and a dated decision to scale, fix or stop. A demo alone does not count as a deliverable.

Why is a demo not enough for a pilot?

A demo shows that an agent can work once. A pilot must show how often it works, at what cost, and against what alternative.

Anthropic’s guidance on agents says to find “the simplest solution possible, and only increasing complexity when needed” (Anthropic, 2024-12-19). The same page says success comes from measuring results and improving the system step by step. A pilot is the place where that measuring starts.

A pilot with no baseline and no written criteria cannot prove value, so it cannot earn a scale decision.

What are the five deliverables?

#	Deliverable	What “done” looks like
1	One scoped workflow	One task, one owner, one user group. Written in a single page.
2	A measured baseline	Time, cost and error rate of the current process, taken from a representative sample.
3	Written success criteria	Pass/fail thresholds, agreed before the agent runs on real work.
4	A small test set	20 to 50 real cases with expected results.
5	A decision rule	A date and a rule: scale, fix or stop.

1. One scoped workflow

Pick one workflow. Name one person who owns it. Woolpert Digital Innovations advises “starting with one workflow” (Woolpert Digital Innovations, undated). Wavect’s scorecard names an owner for every metric (Wavect, 2026-07-16). Both are vendor and consultancy sources. Treat their thresholds as rules of thumb, not standards.

2. A measured baseline

You cannot say the agent is better if you do not know what “before” looked like. Wavect says a pilot should be judged against a measured pre-AI baseline. It also says to time a representative sample, not only the easiest cases. It advises keeping the P95 task time and cost, since an average can hide a slow exception queue.

3. Written success criteria

Write the criteria before the agent touches real work. Wavect says to define them before the pilot begins and to schedule the scale, iterate or kill decision for a fixed date. Include cost. Wavect defines cost per successful action as all operating cost, including human correction, divided by the actions that meet the quality bar.

4. A small test set from real failures

Anthropic’s engineering team says teams often wait for hundreds of test tasks. It calls “20-50 simple tasks drawn from real failures” a great start (Anthropic, 2026-01-09). The same article says early changes have large effects, so small samples are enough to see them.

Hamel Husain makes the same point from the practitioner side: remove friction from looking at data (Hamel Husain, 2024-03-29). Read real outputs. Label them pass or fail. Then write the tests.

5. A decision rule

End the month with a decision, not a mood. Wavect names three outcomes: scale, iterate or kill. Write the decision rule at the start, and put a date on it.

What should you measure in the pilot’s first weeks?

Keep the list short. Woolpert’s first-30-days guide names five items: time savings, task success rate, adoption, accuracy and human corrections, and a weekly reporting snapshot (Woolpert Digital Innovations, undated). It cites 80% task success as a target. That number is the author’s rule of thumb, not a benchmark.

For a fuller list of early signals, see What should you track in the first days of an AI agent pilot?

What should make a pilot stop?

Set stop rules before launch. Wavect proposes three hard gates that override any average score:

1. An unacceptable harm or false-positive event. Pause or stop, even if the economics look good.
2. No audit trail for a high-impact action. Do not raise autonomy until each run can be reconstructed.
3. No defensible baseline or outcome definition. The pilot cannot prove value, so it should not scale.

Anthropic adds a related point for autonomous agents. It recommends “extensive testing in sandboxed environments, along with the appropriate guardrails” (Anthropic, 2024-12-19).

How do you split capability tests from regression tests?

Anthropic’s engineering team describes two kinds of eval. Capability evals ask what the agent can do well. They should start with a low pass rate, which gives the team “a hill to climb”. Regression evals ask whether the agent still handles what it handled before. They should sit near 100%. When a capability eval reaches a high pass rate, it can move into the regression suite (Anthropic, 2026-01-09).

For a pilot, this gives a simple rule. Use the capability set to decide if the agent is good enough. Use the regression set to make sure each fix does not break something that worked.

What does a good end-of-month review look like?

Hold one review meeting. Bring four things: the baseline numbers, the pilot numbers on the same measures, the list of failures the team read by hand, and the decision rule written at the start. Compare the pilot to the baseline, not to a hope. Then apply the rule and record the result.

If the result is “fix”, name the top two or three failure types and add test cases for each. If the result is “scale”, name the next workflow and repeat the baseline step for it. If the result is “stop”, write down why, so the next pilot does not repeat the mistake.

FAQ

Should a pilot try to automate the whole workflow?

No. Start with one narrow slice with a named owner. Anthropic advises using the simplest solution and adding complexity only when needed. A narrow slice also makes the baseline easier to measure.

How big should the first test set be?

Anthropic’s engineering team suggests 20 to 50 simple tasks from real failures as a start. Eugene Yan suggests more for judge calibration: at least 50 to 100 failing cases among 200 or more labeled samples (Eugene Yan, 2025-11). These two numbers serve different goals. The first is for early agent testing. The second is for checking an automated judge.

Is a high accuracy score enough to scale?

No. Wavect says a high average can hide an unacceptable safety, false-positive or audit failure. Check the stop rules, the cost per successful action and adoption before you scale.

Who should read the outputs during a pilot?

People who know the work. Anthropic says teams cannot know if their graders work “unless you read the transcripts and grades from many trials”. A domain expert is the best reader of those transcripts.

How Clarity can help

Clarity by Epistemic Me is an AI-native services firm. It builds AI agents with client teams and runs evals consulting for AI agents, as done-with-you or done-for-you work. The first phase of an engagement is Sprint Zero. In Sprint Zero, the team reviews inputs, establishes a baseline and agrees the first increment. Scope, deliverables and duration are set in the proposal, and engagements start at $100K+. For Dayforce, Lynn Chyi (AI Experience Architect at Dayforce) said the work improved “our human expert alignment scores by nearly 60%” (case study). See /evals/ and /services/, and read how to evaluate an AI agent.

Sources

• Anthropic — Building effective agents (2024-12-19)
• Anthropic — Demystifying evals for AI agents (2026-01-09)
• Hamel Husain — Your AI product needs evals (2024-03-29)
• Eugene Yan — Product evals (2025-11)
• Wavect — AI pilot kill-or-scale scorecard (2026-07-16, updated 2026-09-02)
• Woolpert Digital Innovations — What to watch in the first 30 days of your conversational AI agent (undated)