The Eval Is the Steering Wheel. Your Firm's Judgment Is the Driver.

Your firm's judgment is what makes AI agents good at audit. The eval is how they learn it.

Every audit firm has a way of doing things that isn't fully written down: what counts as enough evidence for a design test, when a blank date is an exception, which documents a reviewer will accept for a given control. New staff learn it from review notes, one engagement at a time.

AI agents have to learn it too. A general-purpose model knows the auditing standards, but not your methodology. When we reviewed cases where our agents and auditors disagreed, about two-thirds came down to firm-specific methodology or evidence the agent had never seen, not to flawed reasoning.

To know whether an agent has picked up your firm's judgment, you need an evaluation, or eval: a set of measurements of how well it does the work. How often does it reach the same conclusion your reviewer did? How often does it link the right evidence? How many exceptions does it flag that aren't real? Each measurement compares the agent's work against answers your experienced reviewers signed off on real engagements.

Evals are the only reliable way to know whether an agent is getting better. A spot check tells you how the agent did on the files you happened to open. An eval covers hundreds of cases, including the hard ones, and shows whether a change that fixed one problem broke something else. Until a change has been run against the eval, whether it's a new model, a rewritten instruction or a new tool, you don't know if it helped.

This year evals became much more valuable. Improving an agent used to mean an engineer reading its mistakes, guessing at a fix and testing it, one change at a time. Now a coding agent can run that loop on its own for hours: run the eval, read the failures, change the agent, run the eval again. Every company has access to the same coding agents. What differs is the eval each one gives them.

Here is what happened when we tried this on one of our audit agents.

Eleven iterations, no human fixes

Every conclusion a Field Agent reaches carries citations that take the reviewer to the exact place in the source evidence. As we wrote in why review is the new prepare (https://www.fieldguide.com/blog/review-is-the-new-prepare-the-auditor-s-job-in-an-agent-first-world), citations are what let an auditor verify agent output in minutes rather than hours.

We were rebuilding the part of our agents that finds and attaches those citations. The old version handed all the evidence to the model in one large request. The new one searches the evidence and reads only what it needs. The new design was more flexible, but it started far behind. Taking its citation accuracy score (F1, which balances finding every citation that should be there against adding wrong ones) as 100, the old version scored 227.

That gap would normally take weeks of engineering. Instead, we built a fixed set of examples from real engagements, scored automatically by an AI reviewer we had checked against expert judgment, and gave a coding agent one goal: raise the score.

In one working session, over 11 rounds of changing something, measuring it and deciding whether to keep it, the agent took the score from 100 to 228. That brought it level with the old version, without adding wrong citations. No one on our team wrote a fix.

It didn't get there in a straight line. Twice it tried ideas that made things worse, and backed them out. One of those was instructive. In round 7, the agent started merging overlapping citations using the same rule the automated reviewer used to score them. Duplicates went to zero and the score dropped, so it backed the change out. A system optimizing against a score will learn the scoring if it can see it, which is why we keep checking the reviewer against people.

Ten examples is a small test, so we treated the result as a hypothesis. When we built a production candidate of the loop and ran it on all 176 examples in our citation set, it still beat the old version.

What the agent learned about audit evidence

The biggest single improvement came from the evidence itself. Most of the citations the agent was missing sat inside screenshots and scanned PDFs, which it couldn't search. It first tried opening images one at a time, which was slow and used up most of its effort. What worked was having an image-reading model write a short description of every image when the evidence is loaded, so the agent can search a screenshot the same way it searches a document.

Much of audit evidence is visual: user access lists pasted as screenshots, scanned bank statements, photographed approvals, settings screens. Anyone who has staffed an engagement has dealt with these. Any system that cites audit evidence has to read images, and we now build that in everywhere.

The second-biggest improvement came from how the agent handled rejected work. Before anything reaches an auditor, our agents check their own work against strict rules. Some runs failed that check several times and ended with no citations at all, even though most of the work was sound. The coding agent changed the process to keep the parts that passed and redo only the rest, rather than throwing out a whole workpaper because one tickmark was wrong.

Both fixes came from the eval showing why an answer was wrong, not just that it was wrong.

The human work moved upstream

No one wrote a fix, but people did real work. Almost all of it went into designing the evaluation and making it fast.

Run through our normal production infrastructure, one evaluation pass took about 50 minutes. Run directly, without the production queues, it took about 8. That's the difference between a few experiments a day and a dozen in an afternoon, and a coding agent can only learn as fast as you can tell it whether it got better. OpenAI reached the same conclusion (https://openai.com/index/building-self-improving-tax-agents-with-codex/) building tax agents with Codex: the loop runs at the speed of the eval, and the agent needs to see why it failed, not just its score.

Every improvement that stuck came from what surrounds the model rather than the model itself. That means the information the agent is given (its context) and the system that runs it (the harness), including how evidence is organized, which tools it can use and how mistakes are handled. When we built a production candidate of the same loop, we reached the same conclusion: what moved the numbers was mostly the plumbing around the model.

This is also why we built Pythia (https://www.fieldguide.com/blog/long-horizon-evals), our framework for evaluating long-running agents without a human in the loop. How quickly and reliably we can measure now sets how quickly the product improves.

The real bottleneck is the signed-off answer

If a coding agent can steadily improve anything you can measure, the limit is what you can measure. In audit, that limit is sharp.

In most AI work, the correct answer for an example (the ground truth) is a label someone can produce in seconds. In audit, it is a conclusion an experienced auditor signed off, applying the firm's methodology to evidence that is often incomplete. An engagement might include the invoices for a test but not the payment records, or a finding might depend on context from last year's file. Building one usable example means finding an engagement where the work, the evidence and the signed-off answer all line up, then having an expert confirm it.

Most don't line up. When our team catalogued candidate engagements for new agents this fall, the share that could become usable examples fell from about 30% to about 8% as we moved into more industry-specific work. That's roughly one in twelve. Labeling each project took around 10 hours of expert time.

Experienced reviewers also sometimes reach different conclusions on the same evidence. As we wrote in Auditing an audit agent (https://www.fieldguide.com/blog/auditing-an-audit-agent), that is why every disagreement between an agent and an auditor gets expert review before either answer is treated as correct. The eval needs the same professional skepticism auditors apply to evidence.

Everyone rents the same models and coding agents. An evaluation built from real agent work, reviewer corrections and practitioner judgment can't be rented. Every correction an auditor makes in review is a potential evaluation example, and turning those corrections into evals is what compounds over time.

What this means for your firm

For firms putting agents to work, the things that make your firm good at audit are the things that make agents good at audit.

Your methodology and past work make agents accurate. Most agent mistakes come from missing firm context, which is why we built Agent Knowledge (https://www.fieldguide.com/blog/agent-knowledge) to ground agents in a firm's own prior-year workpapers.

Your reviewers' corrections teach the agent. Every time a manager corrects an agent's output, that correction says something precise about what good work looks like at your firm. In Fieldguide, corrections are attributed to the reviewer and never overwritten by a rerun.

Your experts become more valuable. As we argued in Review Is the New Prepare (https://www.fieldguide.com/blog/review-is-the-new-prepare-the-auditor-s-job-in-an-agent-first-world), the auditor's job is concentrating into judgment, and defining what good looks like for an agent uses the same skill.

Ask every AI vendor about their evaluations. A demo shows what an agent did once. An evaluation shows whether it keeps improving. Five questions worth asking:

1. How do you measure whether the agent's work is correct?
2. Who built those measurements, and were they practitioners?
3. How does the agent apply our methodology rather than a generic standard?
4. What happens when the evidence doesn't support a conclusion?
5. Can you show results on engagements like ours, not just a demo?

The coding agent is the engine, and everyone can buy the same one. What sets products apart is the steering wheel: an eval that measures what good work looks like, and practitioners whose judgment defines it.