How do you build AI agents that learn from expert feedback?

An AI agent learns from expert feedback through a loop that you build around it. Experts give pass or fail verdicts with written reasons. You turn those verdicts into tests and a grader. You then change the prompt, tools, examples, or stored notes, and re-run the tests. If scores rise and old tests still pass, the agent learned.

What does “learn from expert feedback” mean for an AI agent?

The agents in this guide do not update their own model weights. Their behavior changes because the system around the model changes. That means the instructions, the examples, the tools, and any notes the agent keeps between sessions.

The Reflexion paper shows one version of this idea. It describes agents that improve through “linguistic feedback” instead of weight updates (Shinn et al., arXiv, 2023-03). The agent reflects on feedback in words and keeps those reflections in an episodic memory to guide later attempts. The authors report gains in decision-making, coding, and reasoning tasks.

Reflexion is research, not a product recipe. Its feedback came from scalar scores or free-form text from external or simulated sources. The lesson for a business is simpler. Feedback must be written down, stored, and used on the next attempt. A comment in a chat thread that nobody records teaches the agent nothing.

Which ways can an agent use feedback?

There are four common routes. They differ in speed, risk, and how easy they are to undo.

Route	What changes	Good for	Main risk
Edit the instructions	System prompt or task prompt	Repeated, clear mistakes	Fixing one case breaks another
Add examples	Few-shot cases with expert critiques	Style and judgment calls	Examples go stale
Keep notes in memory	Files the agent reads before a task	Facts about users, policies, past decisions	Wrong or poisoned notes persist
Run a checker before output	A second model call reviews the draft	Output with clear criteria	Extra cost and latency

Anthropic describes the last route as the evaluator-optimizer workflow. One model call generates a response and another gives evaluation and feedback, in a loop (Anthropic, 2024-12-19). Anthropic says it fits when criteria are clear and refinement adds measurable value. It names a good sign: model output improves when a human states feedback.

Anthropic’s memory tool shows the notes route. The agent stores what it learns in files and reads them back in later conversations. The docs list “apply lessons from past interactions, decisions, and feedback to new tasks” as a use case (Anthropic, Claude API docs, accessed 2026-10-30). The same page says memory is client-side. Your application controls where it lives.

Anthropic’s engineering team also describes structured note-taking as a way to persist progress outside the context window (Anthropic, 2025-09-29).

How do you collect expert feedback that an agent can use?

Raw opinions do not work as training signal. Use a fixed process. This checklist follows the method Hamel Husain describes in his LLM judge guide (Husain, 2024-10-29, updated 2026-09).

1. Pick the principal domain expert. Choose the one or two people whose judgment decides what “good” means. Do not use developers as stand-ins.
2. Build a varied set of real cases. Cover features, scenarios, and user types.
3. Collect a pass or fail verdict for each case. Husain warns against 1 to 5 scales because they are hard to act on.
4. Collect a written reason for each verdict. The reason is the useful part. It tells the agent, and later the grader, what the expert noticed.
5. Fix the obvious errors first. Then return to expert review for the harder cases.
6. Group the reasons into failure types and count them. Husain calls error analysis “the single most valuable activity in AI development” (Husain, 2025-03-24). In one example from that guide, a date-handling success rate rose from 33% to 95%.

Give the expert a simple tool. The field guide argues that domain experts are often best placed to improve AI systems, and that teams should drop jargon when they work with them. Read the expert feedback to rubric guide for the next step.

How do you turn feedback into agent changes?

Match each failure type to the cheapest fix that works.

• A rule the agent keeps breaking. Add the rule to the instructions. Add one expert-labeled example next to it.
• A judgment call that needs taste. Add several expert cases with their written reasons as few-shot examples. Husain uses the expert’s critiques as few-shot examples in the judge prompt.
• A fact that changes by user or account. Store it in memory or a retrieval source. Do not hard-code it into the prompt.
• A step the agent skips. Change the tool design or the workflow. Anthropic says agents need “ground truth” from the environment at each step, such as tool results.
• A high-risk action. Add a human approval step. Do not rely on learned behavior.

Change one thing at a time. Then re-run the same cases. If you change three things at once, you will not know which one worked.

What can go wrong when an agent learns from feedback?

Four problems come up often.

The experts change their minds. Shankar and co-authors call this criteria drift. People need criteria to grade outputs, but grading outputs also helps them define the criteria (Shankar et al., arXiv, 2024-04). So treat the rubric as a living document. Re-label a sample of old cases after each major rubric change.

One fix breaks another case. Anthropic separates capability evals from regression evals. Regression evals ask whether the agent still handles tasks it used to handle. They should pass at close to 100% (Anthropic, 2026-01-09). Move every fixed failure into the regression set.

The grader drifts away from the experts. An LLM judge needs checks against expert labels. Anthropic says model graders should be closely calibrated with human experts. Husain tracked agreement across prompt rounds and reached over 90% on one project in three iterations. He also warns that raw agreement can mislead when failures are rare. He recommends reporting true positive and true negative rates separately.

Stored notes become wrong or unsafe. The Anthropic memory docs tell builders to cap file sizes, expire old memory files, and validate every path. They also warn that a bad path can reach files outside the memory folder. Microsoft’s governance guidance asks teams to audit memory stores and logs and to set retention periods (Microsoft, Cloud Adoption Framework, 2026-04-09). Someone must own the memory. Someone must be able to delete from it.

How do you know the agent actually improved?

Use three numbers. Report them after every change.

1. Expert agreement. How often the grader matches the expert on a held-out set of cases.
2. Pass rate on the new failure type. Did the targeted fix work?
3. Pass rate on the regression set. Did anything old break?

Anthropic also advises you to read transcripts. This shows whether a failure is a real agent mistake or a grader that rejected a valid answer. Its blunt advice: “Read the transcripts!”

If your agent also keeps memory, add a fourth check. Run the test set once with memory on and once with memory off. If the scores match, the memory is not helping.

See how to evaluate an AI agent and the LLM-as-judge guide for more on graders.

FAQ

Do AI agents need fine-tuning to learn from feedback?

Not always. The Reflexion paper describes improvement through language feedback without weight updates. Many teams get the first gains from better instructions, expert examples, and a test set. Fine-tuning is a separate choice with its own cost and data needs.

How much expert time does this take?

It depends on your scope, so no single number is honest. Start with a small, varied set of real cases that one principal expert can review. Expand it as new failure types appear. Husain’s process starts with the expert, not with tooling.

Can the agent update its own memory without review?

It can, but you should add controls. Cap memory size, expire old files, log every write, and let a human review changes for high-risk topics. Treat memory as part of the system you test, not as a free side effect.

Where does a human still need to approve actions?

Anywhere an action is costly or hard to undo. Google Cloud’s design guidance describes human-in-the-loop checkpoints for oversight, subjective judgment, and final approval of critical actions (Google Cloud, last updated 2026-05-28).

How Clarity can help

Clarity by Epistemic Me is an AI-native services firm. It runs evals consulting for AI agents, done-with-you or done-for-you. The work covers error analysis, expert rubrics, golden datasets, LLM-as-judge graders checked against expert labels, and regression gates. It also builds AI agents with client teams. Engagements start at $100K+. Scope, deliverables, and duration are agreed in the proposal, and the first phase is Sprint Zero.

Lynn Chyi, AI Experience Architect at Dayforce, said the work improved “our human expert alignment scores by nearly 60%.” Read the Dayforce case study. See /evals/ and /services/ for the offers. For background, read what an AI-native services firm is.

Sources

• Shinn et al. — Reflexion: Language Agents with Verbal Reinforcement Learning, arXiv (2023-03, revised 2023-10)
• Anthropic — Building effective agents (2024-12-19)
• Anthropic — Memory tool, Claude API docs (accessed 2026-10-30)
• Anthropic — Effective context engineering for AI agents (2025-09-29)
• Anthropic — Demystifying evals for AI agents (2026-01-09)
• Hamel Husain — Creating a LLM-as-a-Judge that drives business results (2024-10-29, updated 2026-09)
• Hamel Husain — A Field Guide to Rapidly Improving AI Products (2025-03-24)
• Shankar et al. — Who Validates the Validators? arXiv (2024-04-18)
• Microsoft — Govern and secure AI agents across the organization, Cloud Adoption Framework (2026-04-09)
• Google Cloud — Choose a design pattern for your agentic AI system (2026-05-28)