What are evals for an AI agent, and how do you use them?

Short answer. Evals are repeatable tests that run an AI agent on known cases and check the result. Each case is a real piece of work with an answer a person has already confirmed. You run the evals before anyone relies on the agent’s work, and again after every change of model, rules or access. The company owns its evals, and the operator, the person in charge of the agent, decides what counts as right.

This guide shows what one eval looks like, what to test, where the cases come from, when to run them, who owns them, and how each correction becomes a new case.

Why trying it a few times is not enough

An agent that handled five invoices well may still fail on the sixth. A model can answer the same input differently from one run to the next. A change in one place, such as a new model or a new rule, can also break a step that used to work. A few manual tries cannot show either. Anthropic’s guidance on agent evals makes both points: outputs vary between runs, so each case is run more than once, and a set of fixed cases shows whether the agent “still handles all the tasks it used to”.

Evals also differ from the benchmarks AI vendors publish. A benchmark measures a model on general tasks. Your evals measure your agent on your own work: your suppliers, your accounts, your rules.

What one eval looks like

An eval case has three parts: the input, the known answer, and the check that compares them.

Part	What it is	Example: posting a supplier invoice
The input	A real piece of work the agent receives	An invoice from a known supplier, for office supplies
The known answer	The result a person has confirmed is right	Account, VAT code and cost centre as the bookkeeper posted it last time
The check	What decides pass or fail	The draft entry has the same account, VAT code and cost centre, links to the invoice, and is still a draft waiting for approval

The check looks at what ended up in the system, not at what the agent said it did. An agent can report that a draft was created; the eval checks that the draft exists, with the right fields. Anthropic’s guidance recommends the same: grade the outcome, not the path the agent took.

Four things to test

What to test	The question	Example
The right result	Does the output match the known answer?	The posting matches the invoice: account, VAT code, cost centre, amount
A source on every answer	Does every figure show the record it came from, and does the agent say so when there is none?	The draft links to the invoice; a question about a missing invoice gets “no source found”
The right sign-off zone	Does the step stop where it should?	An invoice from a new supplier waits in “asks first”; a payment always goes to the operator
The forbidden cases	Does the system refuse what the agent must never do?	Asked to pay the invoice, the agent cannot; see guardrails

Test both directions: the cases where the agent should act, and the cases where it should stop or ask. A set that only checks the first can pass an agent that acts on everything.

The three sign-off zones, “acts alone”, “asks first” and “the operator signs off”, are explained in When may an AI agent act on its own?

Where the cases come from

• Work a person has already checked. Last month’s invoices, as the bookkeeper posted them, give known answers without anyone writing new ones.
• The operator’s corrections. Every step the operator had to correct is a case the agent once got wrong.
• The hard cases. Credit notes, foreign currency, an invoice split across cost centres, a supplier who changed name.

Start small and grow the set. Anthropic’s guidance says 20 to 50 simple cases drawn from real failures is a good start. The cases are real company records, so they are kept where the records themselves may be kept; the accountability guide covers the personal data in them.

When to run them

• Before the agent starts work. The evals show whether it is ready to prepare anything at all.
• Before a step moves to “acts alone”. The evals show whether the step passes, case after case.
• After every change. A new model, a changed rule, new access, or a change in a system the agent reads from.
• More than once per case. Because a model’s answers can vary, one pass is not proof.

If a case that used to pass now fails, the change waits until someone knows why.

Who owns the evals

Who	What they do
The company	Owns the evals, kept and versioned with the agent’s rules, so they stay with the company if the agent or its builder changes
The operator	Says what counts as right, adds cases from the work, and sets how often a step must pass before it moves
Whoever builds the agent	Writes the checks, runs the evals after every change, and reports what failed and why
The operator, again	Reads the results and decides; nothing moves to “acts alone” because a score went up

Every correction becomes a new case

1. The operator corrects a step. The agent proposed the wrong account for a supplier.
2. The correction goes into the rules. It is written into the agent’s rules or the company dictionary, a written list of what the company’s words and numbers mean. The model itself does not change.
3. The case goes into the evals. The invoice, with the right answer, becomes a new case.
4. The evals run again. The new case passes, and the old cases still pass.

From then on, the same mistake is caught by the evals before it reaches the operator again. Anthropic’s guidance describes the same loop: failures become test cases, and test cases prevent the same failure returning.

Evals and the operator

Evals check known cases. Live work brings cases nobody has seen yet. That is why a step that acts alone still writes every action to the logbook, the record of what was done, on which sources and who decided, and why exceptions still go to a person. Anthropic’s guidance also places evals beside monitoring and human review, never in place of them.

A passing eval gives the operator evidence. The operator makes the decision.

Sources

• Anthropic: Demystifying evals for AI agents (engineering article, 9 January 2026)