Graders · Effective Agent Evals: Best Practices & Methods

Most agent outcomes need a person or a language model to grade them, and neither is ever 100 percent correct. “Two human scorers usually top out at agreeing around 90% of the time,” Steve Yegge wrote in October 2026 [67]. So good grading comes down to three things: choosing and testing your human graders, using LLMs to extend them, and knowing the best score your domain allows. When a program can check the answer, you may not need either, and the chapter on challenges with benchmarks covers how program graders go wrong.

Human graders set the standard

Why start with people? Your graders can be a mix of people and LLMs, but they have to originate with people. LLM judges and stored answers are checked against human verdicts, so the people you choose set the standard for everything downstream.

Some tasks need a human grader

A person is worth the cost when a program can’t check the answer and a model’s verdict hasn’t been validated. These are the common cases:

Type of task	Who should grade it	Example
A right answer exists, and checking it takes expertise	Domain expert	“Find a court case that supports this claim.”
A right answer exists, and checking it takes careful reading	Trained person	“Summarize this contract, including its termination terms.”
Qualified people can reasonably differ	Domain expert or a panel	“Write a memo recommending one of these vendors.”
A mistake is costly or regulated	Domain expert, often two	Medical advice, or a large refund
Nobody has written the criteria yet	Whoever owns the outcome	The first traces you read for a new agent
The question is whether the work was good enough to use	The person who received it	Whether a developer kept the code

Human graders come from four places

Each source has its own strengths and its own biases:

Source	Examples	Strength	What to watch for
Your own people	Support leads, engineers, product managers	They know your policy and your customers	They may have built the agent or want it to pass
Outside experts	HealthBench’s physicians [6], GDPval’s occupation experts [7]	Deep knowledge of the work	Limited time, and experts can disagree
Crowd workers	Paid annotators on a labeling platform	Fast and cheap for many tasks	Less domain knowledge
Customers	Ratings, surveys, whether they keep the work	They see the real work in real use	The biases described below

Domain experts who do the work themselves are usually the best graders, and the other sources are best measured against them. Non-experts can come close to experts on some tasks. Expert and crowd verdicts can also differ in consistent ways, and the chapter on disagreement covers how to tell.

Customers as graders carry five biases

Customers see the real work, so their verdicts are tempting to use as a score. Those verdicts carry five biases:

Who responds: ratings and tickets come only from customers who act, and that depends on the customer as much as the agent. A customer who got the wrong refund amount writes in, and one who got the wrong return window may never notice.
What they can check: customers can’t grade what they can’t verify, so a convincing wrong answer can pass. Wen et al. had time-limited people check model answers after training on human feedback. The models got better at convincing them but not at the task, and the people’s false-positive rate rose by 24.1 percent on a reading task and 18.3 percent on a programming task [68].
What they prefer to hear: people favor answers that agree with them. Sharma et al. analyzed existing human preference data to see whether it drives sycophancy in AI assistants. They found that a response matching a user’s views was more likely to be preferred, and that people sometimes preferred a convincing, agreeable response to a correct one [69].
What they believe happened: a customer’s sense of whether the agent helped can be wrong. In a 2025 randomized trial of 16 experienced developers, AI tools increased completion time by 19 percent, and the developers estimated afterward that the tools had cut it by 20 percent. Before starting, they had forecast a 24 percent cut [70]. Mozannar et al. had 243 programmers code with seven language models, through autocomplete or chat. The programmers’ preferences didn’t correlate with their actual performance [71].
Who wants the result: when the customer is the company that owns the agent, its experts may favor their own agent, and whoever commissioned it may want a good result. We haven’t seen this measured for agent evals. The closest evidence is from medicine, where studies sponsored by a drug or device’s manufacturer reported favorable efficacy results 1.27 times as often as other studies. That Cochrane review included 75 papers, and its authors suggest an industry bias that standard risk-of-bias assessments can’t explain [72]. METR’s maintainers weren’t told whether a person or an AI wrote each pull request, which removes one route for this bias. The note admits they might still guess from a patch’s style [8].

These biases also spread. If you train or tune on customer ratings, the agent learns to earn the rating, and Sharma et al. found that optimizing against preference models sometimes sacrificed truthfulness [69]. How much weight customer feedback deserves depends on what you’re using it to decide. The chapter on test cases from logs covers how to measure the bias with a random audit.

Human grades depend on how you run the work

The same grader can give different verdicts depending on the setup, and these considerations matter most:

Match the expertise to the task. A support lead can grade a refund conversation, and only a lawyer can grade a citation.
Keep graders independent. Yegge’s design has two people score each case blind, with a third settling disagreements [67].
Hide who produced the work. METR’s maintainers weren’t told whether a person or an AI wrote each pull request [8].
Collect the verdict before showing a model’s suggestion. A suggestion can move a person’s verdict, and the chapter on labeling interfaces covers how to measure that.
Expect grades to drift. Graders tire and standards move, so the chapter on disagreement treats a new grader as a change of instrument.

You can verify that a human grader is good

Credentials alone don’t establish a good grader. HealthBench’s conversation-specific rubrics were created by 262 physicians. HealthBench found a substantial spread in how well individual physicians graded [6]. These checks show who is actually good:

Give a domain quiz before they grade. Candidates grade cases with known answers and have to clear a bar you set, such as 90 percent. Use cases with unambiguous answers that look like the real work [73].
Run a small calibration batch. Graders score the same cases, you go through every disagreement, and you fix the rubric where the disagreement came from it.
Hide known-answer cases in the live work. This measures quality during real grading and catches a grader who has slipped. Rotate those cases, because workers can learn to detect a small fixed set [74].
Have two graders score a sample blind. A good task is one where two domain experts would independently reach the same verdict [75]. If they don’t, the grader or the task needs work.
Give cases back later. A grader who changes their own verdict on a second read is adding noise.
Estimate each grader’s accuracy from their disagreements. The Dawid-Skene model does this without any known answers [76], and the chapter on disagreement covers it.

Once you trust an LLM judge, it can also watch your graders. A grader whose verdicts keep departing from the judge’s is worth a look, though the judge may be the one that’s wrong.

LLMs extend human graders

A language model can grade at a scale no team of people can, and people are what make its verdicts trustworthy. How well it works varies by task. On HealthBench, GPT-4.1 as a grader beat the average physician on five of seven themes [6]. AgentRewardBench had an expert review each of 1,302 web-agent trajectories, and it evaluated 12 model graders. On whole web-agent trajectories, no model grader exceeded 70 percent precision [53].

So most setups combine the two, in one of these ways:

People set the standard, and the LLM scales it. People score a set of cases, and you trust the LLM only where it agrees with them about as often as they agree with each other [67].
The LLM grades everything, and people audit a sample. The audit covers what passed as well as what failed, and it tells you how often the LLM is wrong.
The LLM handles the clear cases, and people take the hard ones. Cases where the model is uncertain go to a person, so expert time goes where it changes the most verdicts.
A program checks what it can, and the LLM reads the rest. On 450 Gaia2 trajectories, an LLM grading the whole run reached 0.72 agreement with people. A program that matched the agent’s write actions and asked a model only about free text reached 0.98 [77].
The LLM suggests, and a person decides. This is fast, and it can pull the person toward the model’s verdict.

A more expensive model doesn’t guarantee a better grader. On PaperBench, an o3-mini grader reached a macro F1 of 0.83 against human grades at $66 per paper, and an o1 grader reached 0.84 at $830 [78]. Macro F1 combines precision and recall into one score, where 1 is perfect.

The right mix depends on the agent and on what a mistake costs. Part 4 covers this in more depth, in the chapters on labeling interfaces, choosing a judge’s dimensions, disagreement and validating judges.

The optimal grader score is below 100 percent

No grader is 100 percent correct, so no agent can reliably score 100 percent on a test that a person or a model grades. The optimal grader score is the best score your grading process can show. It’s set mostly by how often qualified graders agree with each other, and beyond it you can’t tell whether an agent is getting better.

It depends on the domain. Yegge puts human agreement at about 90 percent for contact-center casework, which is a reasonable starting point in many domains [67]. Other studies report lower numbers:

Domain	Who agreed	Agreement
Contact-center casework	Two human scorers	About 90 percent [67]
Preferences between model responses	People comparing responses	Over 80 percent [79]
Knowledge-work deliverables	Occupation experts	71 percent [7]

These tasks differ, so the table shows the range and doesn’t rank the domains. Measure your own number on your own cases, and the chapter on disagreement covers how, including why raw agreement misleads when one verdict is much more common than the other.

Agreement between graders isn’t quite the same as a perfect agent’s score. Suppose two equally good graders with independent errors agree 90 percent of the time. Each one is then right about 95 percent of the time, so a perfect agent would score about 95 percent against either one, and an agent as good as a typical grader would score about 90 (our arithmetic).

This changes how you read a score. At 88 percent on a task where graders agree 90 percent of the time, your eval has little room left to show progress, and an improvement may not register. A score well above the optimal grader score usually means something else is going on, such as an agent that has learned its grader’s preferences, which the chapter on overfitting covers.

You can raise the limit with clearer criteria, better graders and a third person to settle disagreements. You can’t raise it to 100 percent.

Agents and graders both block a perfect score

When your agent misses a case, the cause can be the agent, the grader or the test itself. Your score mixes all three. Take the pipeline question from the chapter on judges:

What’s blocking 100 percent	Example	What fixes it
Agent: a real mistake	It counts a closed-lost deal and reports $4.7M	A better agent
Agent: inconsistency	It reports the correct $4.2M on three runs of four	A more reliable agent, measured over reruns
Grader: an out-of-date answer	A deal closed after the stored answer was written, so the right total is marked wrong	Fixing the stored answer
Grader: a mistake	A reviewer misreads the deal list and fails a correct run	Better graders, or a second grader
Grader: a real disagreement	One reviewer counts only commit-stage deals, and another counts all open deals	A written definition
Test: an ambiguous question	“Pipeline” has two meanings at your company, and the question doesn’t say which	Rewriting the test

Only the first two rows are about the agent. Improving the agent does nothing for the other four.

Published audits show how large the grader side can be:

Graders can mark good work wrong. OpenAI’s audit of the 138 SWE-bench Verified problems o3 kept failing found material defects in at least 59.4 percent of them. OpenAI stopped reporting the benchmark as a result [80].
Graders can pass bad work. METR’s maintainers reviewed 296 agent-written pull requests, and the benchmark’s automated grader scored 24.2 points above them, relative to a baseline of human patches. The same maintainers merged about 68 percent of those human patches [8].
A grader bug can change the winner. Junjie Chen replayed 4,320 episodes of a widely used benchmark and found its scoring code reading the wrong database. Success fell from 94.3 to 65.5 percent in the replay, and a different agent came out on top [81].
The setup itself adds noise. Identical reruns of an evaluation harness flipped 15 to 36 percent of task outcomes on τ²-bench tasks. Gaonkar concludes that single-run comparisons can’t separate the effect of a presentation change from run-to-run variation [20].

Reading a sample of your failures and labeling each one as agent, grader or test shows you which of these you have.

Here’s what to do about it

Test every human grader before their verdicts count.
Measure how often your graders agree before you set a target score.
Check any LLM against human verdicts before it grades on its own.
Sort a sample of failures by cause, so you know whether to work on the agent, the grader or the test.