AOT

Overview Evals Library

Research About Login Contact

Overview Evals Library Research About Login

Effective Agent Evals: Best Practices & Methods

Part 1 Agent fundamentals

1 Outcomes
2 Judges
3 Graders
4 Mistakes
5 Objective function

Part 2 Statistical foundationsComing soon

6 Validity and reliability
7 Statistical methods for evaluation
8 Reliability under repetition and error baselines

Part 3 Constructing test casesComing soon

9 Test cases from application logs and analytics
10 Test cases from benchmarks
11 Challenges with benchmarks
12 Synthetic test cases
13 Holdout sets
14 Overfitting
15 Detecting shifts in traffic and categories

Part 4 Labels and judgesComing soon

16 Labeling interfaces and protocols
17 Choosing the dimensions a judge scores
18 Disagreement among labelers and judges
19 Validating judges and detecting their biases

Afterword: how we apply thisComing soon
Sources

Subscribe for new chapters

Effective Agent Evals: Best Practices & Methods · Chapter 18

Disagreement among labelers and judges

Coming soon

We're releasing the rest of this book over the coming weeks and months. Subscribe for new chapters and research news.

Email

Previous← Cover

© 2026 Mahalanobis Systems