Judges

Judging assesses how an agent reached its outcome, such as its brand safety, efficiency and clarity of explanation. Grading assesses whether the outcome is right. A grader decides whether the outcome is right and a judge assesses how it was achieved, and either one can be a program, a person, a panel or a language model. You need both, and you have to keep them apart. When Gaonkar’s TRACE (October 2026) showed two frontier judges the same τ²-bench records, they disagreed on 57 percent, largely because one graded procedure and the other graded outcome [20].

A correct number can hide a bad run

A run can fail every judge while the final number is correct. In the figure above, you ask your agent “what’s Q4 weighted dollar pipeline?” and the grader checks for the correct number. Each judge in the figure can still find a problem with a run that reaches it:

• Did it include the right deals? A deal wrongly included can offset one wrongly left out, so the total comes out right with the wrong deals.
• Was it efficient in checking? The agent could use 40 tool calls and $3 of model spend where a reference run takes six calls.
• Did it look outside the CRM? The agent could take the figures from a spreadsheet someone emailed. The total matches today, and the spreadsheet will be wrong next quarter.

Published benchmarks show the same gap. τ-bench scores a run by comparing the final database with an annotated goal state [2], so a run can match that state and still break a rule the state doesn’t record. BizAgentBench has 50 tasks built from a payments company’s real procedures, and deterministic checks assess both goal attainment and execution validity. On BizAgentBench’s payments workflow, one model reaches the goal on 42.0 percent of tasks, and reaches it while satisfying the execution constraints on only 32.7 percent [21].

The gap runs the other way too. An agent can follow every rule and explain itself clearly and still report the wrong number, and a judge reading for tone and policy would pass it. Neither verdict alone describes a run.

Grading each step catches bad reasoning

Rewarding only the final answer lets wrong reasoning through. Uesato et al. compared rewarding the final answer with rewarding each step on GSM8K math word problems. Rewarding the final answer reached a similar final-answer error with less labeling. But among solutions with the correct final answer, rewarding each step cut the reasoning-error rate from 14.0 to 3.4 percent [22].

For an agent, the reasoning is the trajectory, the steps and tool calls in a run. A correct $4.2M built from the wrong deals is a right answer with wrong reasoning, and only a judge reading the trajectory can catch it. This assumes you can judge a step on its own. That fails when the right step depends on later ones, or when legitimate routes differ from the reference path.

Judging dimensions vary by workload

What can a judge assess? Each thing a judge assesses is a dimension. These come up across many workloads, and they’re examples, not a complete list:

• Policy adherence: a support agent offers a refund outside the return window.
• Brand safety and tone: a reply disparages a competitor.
• Efficiency: the agent makes 40 tool calls where six would do.
• Clarity of explanation: the agent gives the right total and doesn’t say which deals it counted.
• Side effects and staying within the request: the agent edits a deal’s close date while answering a question about pipeline [8][23][24].
• Following instructions, including implicit ones: you ask for a one-page memo and get five pages [6][25].
• Asking when the request is unclear: “pipeline” could mean two things, and the agent guesses [6][26][27].
• Citing and tracing evidence: a report makes a claim with no source behind it [28][29][30].
• Protecting private data: the agent shows one customer’s details to another [15][31].
• Claims that match what it did: the agent says it changed a card without updating the database [32][33].

Other dimensions depend on the work. These are examples from published benchmarks and studies:

Published work is thin for IDE chat, and we found none for tax documents or for outbound email.

Some outcomes have no single right answer, such as a memo or a research report. The chapter on outcomes shows how to grade those in parts. A judge can still assess how the work was done, such as whether the agent checked primary sources.

Each dimension needs its own evidence and its own way of scoring. A rubric is the written instructions a judge follows:

Grades and judges’ scores have to combine into a decision

How should one grade and several judges’ scores become one result? It depends on what a mistake costs, who reads the result and what you’re using it to decide. The chapter on the objective function covers six strategies.

Recent research measured the judges’ errors

How much error do judges bring to a verdict? It depends on what the judge can see, what it thinks it’s judging, how the policy is worded and how hard it’s pushed.

A judge can’t catch what its input doesn’t show. Mohammadi (August 2026, revised October) built a simulated support desk that injects single-step faults, so every run’s true label was known [61]. A judge given only the request and the final reply flagged 34 to 76 percent of four fault types that leave the reply unchanged. It flagged the matching clean runs at the same rate, because its input was identical in both cases. Told to check each step, the same model flagged every such fault and 0 of 100 clean runs. A reply-only judge sees $4.2M whichever deals went into it, so give your judge the trajectory.

A transcript-only estimate has the same limit. In Anthropic’s samples of conversations from November 2025, the model’s own assessment of task success was 67 percent on the consumer product and 49 percent on the API. As the report says, that’s a model reading a transcript [62].

Two judges can disagree about what they’re judging. TRACE changes one part of an evaluation at a time and compares the paired runs [20]. Each frontier judge stayed consistent when the same trajectory was presented differently, and the two still disagreed. So write down the property each judge is meant to assess.

The policy’s wording moves the verdict. Cao (September 2026) audited two τ²-bench domains and found policies that are silent, ambiguous or contradictory. Affected tasks raised run-to-run inconsistency for every model, and “policy specification quality sets the ceiling on evaluation quality” [63]. Separately, rewrites of a safety policy that kept its meaning flipped up to 9.1 percent of judge verdicts above baseline jitter (Weng, Feng and Xie, May 2026). Between 18 and 43 percent of all observed flips occurred on unambiguous cases under such rewrites [64]. A policy verdict measures your policy text along with your agent.

Scoring from the log beat a holistic read. Qin, Yin and Wang (August 2026) gave partial credit from the execution log, using a checklist built from human-written rules before any trajectory was read [65]. They measured AUC, the chance that a random passing run scores above a random failing one. The checklist reached 0.689 against 0.619 for holistic judging on 4,455 SkillsBench trajectories. It applies the official grader’s verdict in a separate scripted step.

Judges change their verdicts under pressure. Nine frontier judges on fourteen tasks flipped their verdicts 25 to 71 percent of the time under static pushback. Against an adversarial persuader, they flipped 62 to 91 percent of the time (Zhao et al., August 2026) [66]. A changed verdict almost always made the result worse against ground truth. The chapter on validating judges covers how to test for this before you trust a judge.

Putting this all together

Take the pipeline question. That question and the correct number make one test, and a set of tests like it is an eval. Each time the agent answers is a run. The $4.2M on the screen is the run’s outcome, and the CRM queries behind it are its trajectory.

A grader decides whether $4.2M is right. A judge assesses how the agent got there, one dimension at a time, following a written rubric. A judge that’s a language model is an LLM judge. A program, a person or a panel can fill either role. You record both verdicts for every run, and how you combine them depends on your situation.