What should you track in the first days of an AI agent pilot?

In the first days of an AI agent pilot, track six things: the pre-agent baseline, pass/fail results on real cases, human corrections, cost per successful action, latency and token use, and whether people keep using the agent. Turn on logging before the first request, so every run can be read later.

Why does the first week need its own tracking plan?

Early signals decide what you fix next. Anthropic’s engineering team says early changes tend to have large, visible effects, so small samples can detect them (Anthropic, 2026-01-09). You do not need a large dataset to learn a lot in week one. You need a record of what the agent did and a way to say whether it was right.

Hamel Husain puts it as a rule: “You must remove all friction from the process of looking at data” (Hamel Husain, 2024-03-29). A tracking plan serves that rule. It makes sure the data exists and is easy to read.

What are the six things to track?

#	Track	Why it matters	Where it comes from
1	Baseline	Shows what “better” means	Time, cost and error rate of the current process
2	Pass/fail on real cases	Shows if the agent does the job	A small set of real cases with expected results
3	Human corrections	Shows where the agent is wrong	Edits, rejections and escalations by users
4	Cost per successful action	Shows if the economics hold	Model, tool, retry and human review cost, divided by good outcomes
5	Latency and tokens	Shows speed and spend	Request logs
6	Adoption	Shows if people trust it	Weekly active users, repeat use

1. The baseline

Measure the current process first. Wavect says a pilot should be judged against a measured pre-AI baseline, taken from a representative sample and not only the easiest cases (Wavect, 2026-07-16). Wavect is a consultancy, and this is its own framework. It adds that the P95 task time and cost matter, because an average can hide a slow exception queue.

2. Pass/fail on real cases

Start with binary labels. The authors of an O’Reilly series on building with LLMs report that yes/no judgments and pairwise comparisons give more consistent and faster labels than Likert scales (O’Reilly, 2024-05-28). Eugene Yan also recommends binary pass/fail labels for objective criteria (Eugene Yan, 2025-11).

Anthropic suggests “20-50 simple tasks drawn from real failures” as a first set. Run it on every change to the prompt, tools or model.

3. Human corrections

Count each time a person edits, rejects or escalates an agent result. Woolpert Digital Innovations calls accuracy “the make-or-break metric” for a pilot. It suggests tracking where people reject or correct drafts, routed tickets or data pulls (Woolpert Digital Innovations, undated).

Group the corrections by type. The types become the list of failure modes, and the list becomes your next test cases.

4. Cost per successful action

Raw model spend can mislead. Wavect defines cost per successful action as all pilot operating costs divided by the number of actions that meet the quality bar. Its cost list includes model, tool, retry, human correction and incident costs. A cheap agent that needs heavy human repair is not cheap.

5. Latency and tokens

Anthropic’s engineering team lists latency, token usage, cost per task and error rates as metrics you can track over a static set of tasks. If you run on Amazon Bedrock, model invocation logging records the request, the response, and input and output token counts. AWS documents it as disabled by default (AWS, Amazon Bedrock docs, accessed 2026-10-30). Turn it on, or set up your own logging, before the pilot starts.

6. Adoption

Woolpert’s guide suggests checking whether people use the agent regularly and whether use rises week over week. If experts avoid the tool, accuracy numbers will not save the pilot.

What should you log on every run?

Log enough to rebuild any run later. A practical minimum:

1. The input and the final output.
2. Each tool call and its result.
3. The model and prompt version.
4. Token counts and elapsed time.
5. A user action: accepted, edited, rejected or escalated.
6. A case ID that links the run to your test set.

Wavect gives a reason. It says that without an audit trail for a high-impact action, you should not raise the agent’s autonomy until each run can be reconstructed.

Check what your logs contain. The AWS page says invocation logs hold full request and response bodies. Those bodies may include sensitive data, so set access and retention rules for the log store before you turn logging on.

How do you read the data in the first days?

Read it by hand. Anthropic’s engineering team says teams cannot know if their graders work “unless you read the transcripts and grades from many trials”. Reading shows whether a failure is a real agent mistake or a grader that rejected a valid answer.

A simple routine for each day:

1. Pull a sample of runs, including failures and corrections.
2. Label each one pass or fail.
3. Write one line on why each failure happened.
4. Group the lines. Fix the biggest group first.
5. Add each fixed failure to the test set.

The O’Reilly authors describe an “intern test”. If an average college student in the relevant major could do the task quickly with the same input, examine the data and ask what the model does wrong.

What should you not track yet?

Skip vanity metrics in the first days. Total chats, thumbs-up counts with no sample size, and average scores with no failure review all look good and teach little. Do not set a final go or no-go number from three days of data. Wavect says to schedule the scale decision for a fixed date, set at the start. For what the pilot should deliver by the end of the first month, read What should an AI agent pilot deliver in the first month?

FAQ

How many cases do I need in the first days?

Start with 20 to 50 real cases, per Anthropic’s engineering team. That is enough to see large effects. Add cases every time you find a new failure.

Do I need an LLM judge in week one?

Not always. Start with human labels and simple checks. The O’Reilly authors note that LLM-as-judge is not a silver bullet, and that classifiers can sometimes be more accurate and cheaper. If you add a judge, check it against human labels. See the LLM-as-judge guide.

What is a good task success rate?

There is no standard number. Woolpert cites 80% or higher as a target for its conversational agent guidance. Treat that as one author’s rule of thumb. Set your own threshold from your baseline and the cost of an error.

Who should label the early runs?

People who know the work. A domain expert can say whether an answer is right. A generic reviewer often cannot.

How Clarity can help

Clarity by Epistemic Me is an AI-native services firm. It builds AI agents with client teams and offers evals consulting for AI agents, done-with-you or done-for-you. Its Sprint Zero phase reviews inputs, establishes a baseline and agrees the first increment. Engagements start at $100K+, and scope, deliverables and duration are agreed in the proposal. See /evals/ or /services/, and read how to evaluate an AI agent.

Sources

• Anthropic — Demystifying evals for AI agents (2026-01-09)
• Hamel Husain — Your AI product needs evals (2024-03-29)
• Eugene Yan — Product evals (2025-11)
• O’Reilly Radar — What we learned from a year of building with LLMs, Part I (2024-05-28)
• Wavect — AI pilot kill-or-scale scorecard (2026-07-16, updated 2026-09-02)
• Woolpert Digital Innovations — What to watch in the first 30 days of your conversational AI agent (undated)
• AWS — Monitor model invocation using CloudWatch Logs and Amazon S3 (accessed 2026-10-30)