Why we report review rate, not accuracy
A single accuracy number hides the only thing a finance team feels: how often a person has to step in. Here is the metric we publish instead.
Tomas Reiner, Co-founder, CTO
· 5 min read
When we started, every document-AI vendor we looked at led with an accuracy figure. Ninety-eight percent, ninety-nine, sometimes more. We nearly copied the habit. Then we sat with an accounts payable team for a week and watched what accuracy meant in practice.
The number that hides the work
Accuracy is computed over fields. A typical invoice has thirty. At 98 percent per-field accuracy, the chance that a thirty-field invoice is entirely right is about 55 percent. Nobody who processes invoices experiences “98 percent”. They experience “almost every second invoice has something wrong with it”, and they conclude the tool is useless, because they cannot tell which half.
That is the real problem. The error rate matters less than whether the system knows where its errors are.
What a team actually feels
A finance team feels three things:
- How many documents a person has to touch.
- How long that touch takes.
- Whether a mistake ever reaches the ledger.
None of these is accuracy. The first is a review rate: the share of documents, or fields, held for a human. The second is time to resolve in the review lane. The third is the escaped error rate: wrong values that were not held and were later corrected.
What we publish instead
Inside every workspace, the dashboard leads with those three numbers for your own documents. Review rate tells you how much work the system is leaving you. Escaped errors tell you whether the threshold is set where you want it. And the two move against each other, which is the point: you choose the trade.
A team that wants nothing wrong in the ledger sets a high threshold, accepts a review rate near 15 percent, and sees escaped errors close to zero. A team processing low-value documents lowers the threshold and takes a review rate of 3 percent with a small, known leak.
Why it changes how we build
Optimising accuracy rewards a model for being bold. Optimising review rate at a fixed escaped-error budget rewards being honest: a calibrated score, a clear reason for every hold, and a review lane fast enough that the trade is cheap. Most of our engineering this year has gone into the second list.
If a vendor gives you one accuracy number and no way to see what it leaves for your team, ask for their review rate on your documents. It is the figure you will live with.