Outcomes

Every agent does a different job, so each one has a different definition of a good outcome. A support agent changes a database record, a coding agent writes a patch, a research agent writes a report and a sales agent tries to get a reply. This chapter sorts those differences across common workloads, so you can place your own agent and see what kind of test each of its outcomes needs. An outcome is the end result a run produces, and a grader is whatever decides whether that outcome is right.

Three dimensions describe an outcome

What separates one agent’s outcome from another’s? We sort outcomes on three dimensions:

• What the agent produces: a change to a system, an answer, a document or file, a conversation, or a decision.
• What kind of answer a test can have: verifiable, objective or subjective, often with more than one right answer.
• Who should grade it: a program, a trained person, a domain expert, the person who receives the work, or a language model.

We built this sorting from the benchmarks and production studies cited below. It isn’t a standard taxonomy.

One agent usually has several outcomes, and each one sits in a different place on these dimensions. A coding agent’s patch can pass the repository’s tests, which a program checks in minutes. A maintainer’s decision to merge the same patch is a person’s call, and it can take days.

Agents produce five kinds of output

What does a test actually look at? It depends on what the agent produced, and there are five kinds of output:

• A change to a system: a refund row, a closed ticket or a deployed config. You test it by reading the system afterward.
• An answer: a value, a label or an extracted field. You test it by comparing it with a reference.
• A document or file: a patch, a memo or a spreadsheet. You test it by running it or reading it.
• A conversation: a chat or a voice call, where the exchange is part of the product. You test it by reviewing the transcript or the audio.
• A decision: approve, deny, escalate or decline to act. You test it by comparing it with the decision a qualified person would have made.

Escalating and declining to act count as outcomes. If your test has no expected result for them, it will score them as failures or as successes without anyone having decided which.

Answers are verifiable, objective or subjective

Can a program grade this outcome, or does it take a person? That depends on which of three kinds of answer the test has. Any of the three can have more than one right answer.

Verifiable answers can be checked by a program

An answer is verifiable when a program can return the verdict, either by comparing the outcome with a stored answer or by applying a rule. “Tell me a prime number” has endless right answers, and a program can still test whichever one the agent gives. SWE-bench works by rule: it runs the repository’s tests, so any patch that passes counts [1]. τ-bench compares the final database with an annotated goal state [2][3]. GAIA and BrowseComp choose questions with one short answer, so a stored answer works [4][5].

Objective answers need someone to read the work

An answer is objective when it’s right or wrong, but no program you could practically write can tell which. “Find a court case that supports this claim” has several right answers, and a lawyer has to read the case to know whether it does. Two qualified graders given the same written criterion should reach the same verdict.

HealthBench, a benchmark of language models on health conversations, tests this way. Its 262 physicians wrote 48,562 criteria specific to each conversation, and a grader marks each criterion as met or not met [6].

Subjective answers depend on the grader

An answer is subjective when qualified people can reasonably disagree, such as whether a memo is persuasive or which of two reports is better. No answer exists until the grader gives a view. GDPval is a benchmark of AI models on real work from 44 occupations, across the top 9 sectors of U.S. GDP. In GDPval, occupation experts compared deliverables in pairs and agreed with each other 71 percent of the time [7].

You can turn part of a subjective question into an objective one by writing a criterion. “Is this memo good?” is subjective, and “Does it state its recommendation in the first paragraph?” is objective. So the kind of answer depends on the question you ask, and the chapter on choosing a judge’s dimensions covers how to write those questions.

Most tasks have more than one right answer

A test that compares the output with one stored answer marks every other right answer wrong. So test for what makes an answer right. These tasks show the range:

A research task mixes all three

A research report usually has no single right answer, so grade it in parts. A specific fact is verifiable if someone wrote it down in advance. Whether each cited source supports its claim is objective, because a reader can check it. Whether the report is useful or well argued is subjective, so it goes to an expert or to the person who asked for it. This is why GAIA and BrowseComp ask for one short answer, and why GDPval has experts compare deliverables.

Each kind of answer gives a different test result

Each kind of answer has a best grader

Who should grade an outcome? That depends on the kind of answer and on your situation, and each grader has its own strengths.

Domain experts are the most accurate way to grade. In many cases they can also be the most scalable, because a small set of their verdicts can calibrate a language model that grades the rest.

The person who receives the work can also act as the grader, through what they do next. A developer keeps or discards the code, a customer calls back or doesn’t, and a prospect replies or doesn’t. You don’t ask them for a verdict, because you can read it from your logs.

A language model can stand in for a person once you’ve checked it against that person’s verdicts, and the chapter on validating judges covers how. Language models can also create additional test cases for fine-tuning and other data-intensive optimization methods, which the chapter on synthetic test cases covers.

Common workloads differ on all three dimensions

This table places ten common workloads on the three dimensions. A workload with two rows has two outcomes to test.

Verifiable outcomes are the fastest and cheapest to test, and published evaluations lean toward technical measures. A review of 84 agent-evaluation papers from 2023 to 2025 found technical metrics in 83 percent, and only 15 percent covered both technical and human dimensions. Human-centered assessments appeared in 30 percent [19].

Outcomes a person decides often sit outside your system. CRMArena-Pro tests CRM tasks across sales, service and quoting [15], but no public benchmark we know of tests whether a prospect replies or a deal closes. You can check that your agent sent the email, but the meeting it booked sits in someone else’s calendar.

Here’s what to do about it

• List every outcome your agent produces, including escalating and declining to act.
• Create a grading strategy for each outcome.
• Test for what makes an answer right whenever more than one answer can be right.
• Rewrite subjective questions as objective ones where you can, using criteria two graders would apply the same way.
• Test at least one outcome a person decides, by having experts grade a sample or by tying commits, callbacks and chargebacks back to runs.

Further reading

• Whitfill, Wu, Becker and Rush (2026), “Many SWE-bench-Passing PRs Would Not Be Merged into Main.” Four maintainers from three SWE-bench Verified repositories reviewed 296 agent-written pull requests that had all passed the benchmark’s tests. They would have merged about half, so passing the tests and earning a merge are separate outcomes [8].
• Lei et al. (2024), “Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows.” They built 632 text-to-SQL workflow problems from enterprise database use cases. The benchmark compares what a SQL query returns instead of its text, so different queries with the same result all pass [10].
• Baumann et al. (2026), “SWE-chat: Coding Agent Interactions From Real Users in the Wild.” They collected almost 18,000 real coding agent sessions from open-source developers, using public repositories. They found that 59.4 percent of agent-produced code ends up in the user’s commit, which is a verdict you can read from a log [9].