Mistakes

Some of your agent’s mistakes cost far more than others, so counting them equally tells you little. This chapter shows how to measure how often each kind of mistake happens and how to put a cost on it.

A wrong next step can cost more than a wrong report

Say your CRM agent reports on deals and recommends next steps. It can make a mistake in either job.

A mistake in a report is usually cheap. If the agent says Q4 pipeline is $4.7M when it’s $4.2M, someone checks the CRM, corrects the number and loses an hour.

A mistake in a next step can be far more expensive. Say a deal is out for contract, and the agent reads it as lost and tells the rep to ask the buyer for a closed-lost interview. A buyer who is about to sign gets a message saying you’ve written them off, and that can put the whole deal at risk.

An accuracy score counts these as one mistake each, but one costs 200 times as much as the other (our arithmetic):

To see that difference in your own evals, you need to know how often each kind of mistake happens and what each one costs.

Four rates show how often each mistake happens

How often does your agent get each kind of case right? Say you ask it for a decision-maker report: for each contact on your open deals, is this person a decision maker?

The CRM makes this harder than it sounds, because the fields aren’t consistent. One rep enters “VP Sales” and another enters “Head of Revenue.” Some titles are blank, with “signs the contract” buried in the notes, and others are out of date because the person has changed jobs.

Every verdict is one of four results:

Four measures come from those results:

• True positive rate (TPR): of the real decision makers, the share the agent names.
• True negative rate (TNR): of the other contacts, the share the agent leaves out. A low TNR sends your reps to people who can’t sign.
• Recall: another name for TPR. It answers how many of the real cases the agent caught.
• Precision: of the people the agent names, the share who truly are decision makers. It answers whether you can believe the report.

Say 100 of 1,000 contacts are decision makers. Agent A names 90 of them and also names 45 people who aren’t:

So an agent that is 94.5 percent accurate produces a report where one name in three is wrong.

TPR and TNR describe how the agent handles each group of contacts. Precision also depends on how common decision makers are. If only 20 of 1,000 contacts are decision makers, the same TPR and TNR give a precision of 27 percent (our arithmetic). The chapter on validating judges covers how to estimate these rates from a sample.

A cost matrix adds what each mistake costs

A cost matrix adds a second layer on top of those rates. The rates say how often each result happens, and the matrix says what each result costs.

To fill it in, write down who absorbs each result, what they do about it, and what that costs in a unit your organization already uses. Include the cost of checking and fixing as well as the mistake itself. For the decision-maker report, it might look like this:

To get a total, multiply each result’s count by its cost and add them up. Here’s Agent A against an Agent B that names someone only when it’s sure:

Both agents are 94.5 percent accurate, and Agent B costs almost five times as much (our arithmetic). Agent B also has the more believable report, and it’s still the worse agent at these costs.

Costs change with the situation

A cost matrix isn’t fixed. The same mistake can cost very different amounts, for several reasons:

• The size of the case: a missed decision maker costs more on a $1M deal than on a $10,000 one.
• The stage of the work: the same miss costs little on a deal in early talks and a lot on a deal out for contract.
• Whether someone catches it: if a rep checks the report before acting, the mistake costs a minute. If the agent acts on it directly, it costs the full amount.
• The mix of cases: a contact list with more decision makers gives the agent more chances to miss one, so the total moves even when the rates don’t.
• Who you ask: a sales leader and a finance lead may price the same mistake differently. Record a contested cost as a range, and if your decision changes inside that range, the business has to make the call openly.
• Whether it can be priced at all: some mistakes are ones policy forbids, so they get a hard limit instead of a price.

Put the matrix in your eval report, along with where each number came from. The chapter on the objective function turns these costs into a single score and into decisions.

Further reading

• Turney (2000), “Types of Cost in Inductive Concept Learning.” Turney separates misclassification costs from the costs of tests, intervention, computation and human-computer interaction. For an agent, the test cost is a grader call or a human review, and the intervention cost is an escalation [83].
• Zadrozny and Elkan (2001), “Learning and Making Decisions When Costs and Probabilities are Both Unknown.” They treat costs that differ by case as unknowns to estimate alongside the probabilities. This fits a mistake such as a wrongly approved refund, which costs the refund amount [84].
• Kang and Mussmann (2026), “Instance-Level Costs for Nuanced Classifier Evaluation.” They weight each error by a per-example cost, taken from sources such as annotator vote margins. Models with a 5 percent error rate can reach 1.8 percent normalized excess cost, because most errors fall on ambiguous, low-cost examples [85].