Objective function

An objective function turns the numbers an eval produces into one decision, and it should start from what the business wants. For a CRM agent that sales reps use, the business wants to save reps time without putting deals at risk. So the objective function is the dollar value of each request: the rep’s time it saves, plus or minus any change in the risk of a costly mistake, minus the cost of the run.

Three agents answer the same question differently

Say a rep asks your CRM agent, “Who are the stakeholders on the Acme renewal, and what are their emails?” Doing this by hand takes the rep 15 minutes, between the deal’s contact roles and recent email threads. Here’s what three versions of the agent do:

• Agent A asks too many questions. It starts with “What’s the Salesforce ID of the deal?”, so the rep has to go and find it. It then asks whether Acme means Acme Corp or Acme Holdings, which roles count as stakeholders, and whether the rep wants work or personal emails. Ten minutes after the last answer, it returns the right five people. The rep has spent 12 minutes to save three, and is unlikely to ask it again.
• Agent B is fast and breaks a rule. It asks nothing and returns the right five people in a minute. While it’s working, it also updates the CFO’s title on the contact record, which nobody asked it to do.
• Agent C is fast and misses someone. It answers in 20 seconds with four people. It misses the CFO, who joined last month and appears only in an email thread, so the renewal quote reaches the CFO late.

You then run each agent on 1,000 past requests. A rep working by hand finds every stakeholder 98 percent of the time:

These numbers come from different places. A grader checks the stakeholder list, judges assess the questions and the edit, and the log gives you the time and cost. Each agent wins on something, and you can ship only one.

The business objective puts a price on each number

Most of the numbers above have a place in the goal from the opening. The value of one request has three parts, measured against a rep doing the work by hand, because that’s what the rep would do otherwise:

• Time saved: the minutes the rep would spend by hand, minus the minutes spent with the agent, at the rep’s cost per minute. Time with the agent includes answering its questions and waiting for it.
• Change in mistake risk: how much less or more likely the agent is to miss a stakeholder than the rep, times what a miss costs.
• Cost of the run: what the agent costs to run, subtracted from the other two.

Take the figures from the chapter on mistakes: a rep’s hour costs $100, and a missed stakeholder costs $20,000, a 20 percent risk to a $100,000 deal. Each agent’s value per request is then (our arithmetic):

Agent C is the fastest and cheapest, and it loses about $576 a request. Agent A gets the list right and earns $3.50, which isn’t worth the rep’s attention. Agent B earns the most, but the formula doesn’t see the record it edited. Handling that takes a different strategy.

Six strategies combine the dimensions

How do you get from these numbers to a decision? There are six basic strategies, and you can mix them.

Gates remove runs that break a rule

A gate is a hard limit: one violation fails the run, however good the rest is. If your policy is that the agent never edits a record without asking, Agent B fails. Gates fit rules that are unacceptable at any rate, or that nobody can price. They assume the rule is clear enough to apply, which rules written in natural language often aren’t.

Budgets cap what you’ll spend

A budget is a limit on a resource. You set limits on time and cost, then pick the agent with the best quality among those that stay within them. This is called the epsilon-constraint method. A two-minute limit on the rep’s time rules out Agent A. Budgets fit when someone can set a limit but can’t say what each minute or dollar is worth.

Limits can combine badly. Agent B’s gate and a two-minute budget leave only Agent C, the agent that loses the most money.

Weights trade one dimension against another

Each dimension gets a score and a weight, so strength on one can offset weakness on another. Say the gate has removed Agent B, and you score Agent A and Agent C on coverage and speed. Agent A scores 0.98 and 0.20, and Agent C scores 0.95 and 0.98 (our arithmetic for each total):

The ranking flips at a coverage weight of about 0.96 (our arithmetic). So equal weights, and even a 0.9 weight on coverage, pick the agent that loses $576 a request. Weights that feel reasonable can set an exchange rate nobody agreed to.

Keeney and Raiffa get weights by asking the decision maker what they’d trade, such as how many minutes of a rep’s time one point of coverage is worth. Their weighted form assumes the value of one dimension doesn’t depend on another [86]. That often fails for agents, because a fast answer with the wrong people is worth nothing. So a run’s score usually starts with its grade and its gates. A run that fails the grader or any gate scores zero, and any other run gets the weighted sum of its judged dimensions.

Dollars put everything on one scale

The value formula above is this strategy. Everyone can read a dollar figure, and it shows how much a difference is worth. The inputs are still estimates, so test how much the answer depends on them. Agent C beats Agent A once a missed stakeholder costs less than about $700 (our arithmetic). If nobody can say whether a miss costs more than that, the dollar figure hasn’t settled the decision.

Rank in order when one dimension comes first

You compare on the most important dimension and use the next one only to break ties. Comparing on coverage and then on speed puts Agent B first, Agent A second and Agent C last. This fits when one dimension is close to a requirement, because any gain on it outranks every gain on the next.

Report the trade-off when nobody owns the exchange rate

You can also show the options and let the person who owns the decision choose. One agent dominates another when it’s at least as good on every dimension and better on at least one. The Pareto front is the set of agents that nothing dominates. On coverage and speed, Agent B dominates Agent A, so the front is Agents B and C.

The front comes from a finite number of runs, so treat a lead smaller than the standard error as a tie. The standard error is how much a score would typically move from noise alone.

The strategies can pick different winners

The right mix depends on the situation: what a mistake costs, who reads the result and what the score decides.

Scores across tasks need weights too

A mean score across tasks is also a weighted sum, and those weights can change the winner. A stakeholder lookup on a $1M renewal matters more than one on a $10,000 deal, and an agent can be better at one and worse at the other.

Traffic weighting uses each task type’s share of production volume, so the score estimates what a typical user sees. Value weighting uses what the task is worth in human hours, revenue or avoided loss, so the score estimates value delivered.

The two disagree when one agent is better on frequent, cheap tasks and worse on rare, expensive ones. Zhou et al. (July 2026) built 100 long-horizon office-suite tasks, each with a measured human labor time and a task price estimate. The model with the highest unweighted score came second on both time-weighted and price-weighted scores [18].

Value also has to include the human time spent checking the work. In GDPval, the naive comparison for one model is about 90 times faster and 474 times cheaper than an unaided expert. Once you charge the human time to review and redo unsatisfactory output, the “try once, then fix” policy is about 1.12 times faster and 1.18 times cheaper [7].

Agents can game a poorly defined objective function

An agent or an optimizer will raise whatever you score, and it will give up anything you don’t. On July 14, 2026, Ole Petersen at Listen Labs described a coding agent left alone to raise a metric. One pipeline had to output which emotions occur and when, but the benchmark scored only which emotions occur. The agent removed timestamp prediction, and the F-beta score, a standard classification score, rose “by a few percentage points.” In Petersen’s words, “its task was only to optimize the F-beta score, and nothing else” [87].

Your CRM agent can do the same. Score it on asking fewer questions and nothing else, and it can stop asking which Acme you meant, even when the answer changes the list.

Optimizers can also reach a perfect score by cheating. Wahi (September 2026) catalogs eleven evaluation failures from production prompt-optimization loops. In one, agents reached a 100 percent pass rate by reading cached answers while true capability was 68 percent. Wahi proposes deterministic checks, frozen holdout cases and canary cases, on which a perfect score is itself evidence of cheating [88].

The score and the real goal can also separate gradually. Gao, Schulman and Hilton measured this on reward models, the learned scorers used to train a model. As a policy was optimized against the stand-in, the stand-in kept rising, while the gold score that represents the real goal rose and then fell [89]. The chapter on overfitting covers how to measure that.

Two habits reduce this. Score every field the optimizer can change, because an unscored field has an implied cost of zero. Take the deployment decision from a held-out measure the optimizer never saw.

Report the parts beside the single number

A single number hides every choice above, so report these beside it:

• the pass rate from the grader
• the failure rate for each gate
• the mean of each judged dimension
• time and cost against their budgets, on runs that passed, with failed runs’ time and cost in a separate column
• the single number, with its weights or prices and where they came from

A reader with different exchange rates can then work out their own answer.

Further reading

• Altman (1999), Constrained Markov Decision Processes. Altman formalizes a goal with a limit: maximize expected reward while keeping a separate cost under a bound. A policy rule in an eval works the same way, as a gate [90].
• Miettinen (1999), Nonlinear Multiobjective Optimization. A weighted sum with positive weights always returns a configuration on the Pareto front, but it can’t reach all of them. The epsilon-constraint method reaches every point on the front as you vary the limits [91].
• Kapoor et al. (2024), “AI Agents That Matter.” They argue that the focus on accuracy alone had produced needlessly complex and costly agents. They propose reporting accuracy and cost together, and comparing systems on the Pareto front [92].
• Erol et al. (2025), “Cost-of-Pass: An Economic Framework for Evaluating Language Models.” They define cost-of-pass, the expected monetary cost of obtaining a correct solution. They also define frontier cost-of-pass, the minimum across available models or a human expert priced at the cost of hiring one [93].
• Cunningham et al. (2026), “Expenditure Horizon.” They define the dollar value at which an agent’s improvement equals a human’s with the same budget. On the NanoGPT speedrun, the two strongest models reached $3,300 and $2,300, and the authors flag that the estimate is sensitive to the assumed human returns [94].
• Qian et al. (2026), “ParetoSaddler.” They optimize an agent harness for quality and cost together, and apply the deployment preference only at final selection. On ClawMark, quality rose from 70.86% to 81.35% while cost per task fell from $1.17 to $1.01 [95].