Mahal Systems Effective Agent Evals: Best Practices & Methods
This guide covers the outcomes agents produce, how to judge and grade them, what mistakes cost, how to build an objective function, the statistics that say when a score can be trusted, where test cases come from, and how to make human labels and model judges reliable.
runsjudge score →a person said faila person said passthresholdTNRTPR The first five chapters, on agent fundamentals, are out now. We’ll release the rest of the book over the coming weeks and months.
Research labs spend billions of dollars a year making their models better. Anthropic spent $9.74 billion on R&D in the first half of 2026, and OpenAI spent $19.2 billion in 2025 (R&D World). The investments that labs are making in training their most compute-intensive models have grown about 2.4 times a year since 2016 (Cottier et al.). You’d never imagine building software without extensive tests, and the same is true of an agent built on these models. The number one thing that needs to happen before you can build a great agent is a material investment in testing and evaluations. This guide covers how to make that investment.
An effective evaluation can detect regressions, show you when it’s safe to upgrade a model, optimize your agent, and supply the data for things like fine-tuning models. This guide collects the last 12 months of best practices from research and industry. It joins them with best practices from statistics, machine learning and data science that have existed for decades.
The book has four parts. Agent fundamentals defines the terms: the kinds of outcome an agent can have, across the types of agent in production today; judging, which assesses how an outcome was reached and the qualities a grader can’t see; graders, where they come from and how they go wrong; the cost of each kind of mistake; and how to build the objective function your eval reports. Statistical foundations (coming soon) covers what a score means (validity and reliability), how to estimate and compare scores, and reliability under repetition. Constructing test cases (coming soon) covers where cases come from (logs, benchmarks, generators), how to keep the set from being learned (holdouts and overfitting), and how to tell when traffic has shifted under it. Labels and judges (coming soon) covers the human side of grading: labeling tools, what a judge should score, what to do when raters disagree, and how to validate the judges.