Evals and monitoring
Evals are repeatable tests that run an agent on known cases, with answers a person has already confirmed, and check the result. Monitoring watches the work that runs on its own, logs every run and sends errors to a person. We build both, so your team can decide when a repeated job is ready to run on its own, and see what it does once it runs.
This is for you if
- An agent handled a few invoices well, and nobody can say whether it will handle the next one.
- You would like a repeated job to run on its own, but nobody can show that it is ready.
- A job that runs on its own stopped, and nobody noticed until a figure was wrong.
What you leave with
- The result
- Tests and monitoring your team can read. The eval cases, the checks and the monitoring settings are files and code in your repository, kept with the agent's rules. You decide where the logs and alerts are kept.
- Cases from your work
- Work a person has already checked, your team's corrections and the hard cases, each with the confirmed answer and the check that compares them. The check looks at what ended up in the system, not at what the agent says it did.
- Monitoring and alerts
- Every run of the work that runs on its own is watched and written to the logbook. When a run fails or meets something it does not expect, it stops instead of guessing, and a person is told.
- A decision on record
- The evals give the person who approves evidence. That person decides when a job runs on its own, and the decision is written to the logbook.
What we do, what you do
| We | You |
|---|---|
| Collect the eval cases with you: work a person has already checked, your corrections and the hard cases. | Say what counts as right, and add cases from your own work. |
| Write the checks, including the cases where the agent must stop or refuse, and run each case more than once. | Decide how often the cases must pass before the work runs on its own. |
| Rerun the evals after every change of model, rules or access, and report what failed and why. | Read the results and decide. Nothing runs on its own because a score went up. |
| Set up the monitoring: every run logged, an alert when a run fails, exceptions sent to a person. | Check the results, and approve the work again when something changes. |
Worth knowing
Automation is earned by the work itself. A job is ready to run on its own when it repeats, is written down, its exceptions are known and one person owns the result. The evals show that it passes, case after case. But evals only cover the cases you know, and live work brings cases nobody has seen. So work at the stage "It runs on its own, and you check the results" is still monitored and logged, and four kinds of decision always go to a person: money leaving the company, anything going outside the company, a commitment and an unclear rule. If something changes, it comes back to you for approval.
Your questions
- What does it look like in practice?
- At DuffBeauty, a beauty brand, the data pipelines pull the same data every night. That job repeats and is well described, so it runs on its own and is monitored, and when a source changes, the pipeline reports it instead of guessing. The invoice reader checks its own output: a document that fails a check is flagged with the reason and comes back to the bookkeeper.
- Why not just try the agent a few times?
- An agent that handled five invoices well may still fail on the sixth. A model can answer the same input differently from one run to the next, and a new model or rule can break a step that used to work. Evals run the same known cases every time, so a change that breaks something shows before it goes live.
- When do the evals run?
- Before the agent prepares work for approval, before repeated work runs on its own, after every change of model, rules, access or a system the agent reads from, and on a schedule once the work runs. If a case that used to pass now fails, the change waits until someone knows why.
- What happens when a run fails?
- It stops instead of guessing, the run is logged, and a person is told. In the guide to fetching invoices from vendor portals, a run that meets a page it does not expect saves a screenshot and a trace, and an alert goes out. If something changes, the job comes back to you for approval.
- Who owns the evals?
- You do. The cases and checks are kept and versioned with the agent's rules in your repository, so they stay with you if the agent or its builder changes. For our part, you pay a fixed price for the result, never for the hours Gathena's agents work; ongoing work is a fixed monthly fee.
- Do evals replace the person who approves?
- No. A passing eval gives that person evidence, and the person decides. When someone corrects the agent's work, the correction goes into the rules and becomes a new case, so the same mistake is caught before it reaches your team again.