Mahal Systems

Effective Agent Evals: Best Practices & Methods

Sources

  1. Jimenez, Carlos E. et al. “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” arXiv:2310.06770, October 2023. https://arxiv.org/abs/2310.06770
  2. Yao, Shunyu, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.” arXiv:2406.12045, June 2024. https://arxiv.org/abs/2406.12045
  3. Barres, Victor et al. “τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment.” arXiv:2506.07982, June 2025. https://arxiv.org/abs/2506.07982
  4. Mialon, Grégoire et al. “GAIA: a benchmark for General AI Assistants.” arXiv:2311.12983, November 2023. https://arxiv.org/abs/2311.12983
  5. Wei, Jason et al. “BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents.” arXiv:2504.12516, April 2025. https://arxiv.org/abs/2504.12516
  6. Arora, Rahul K. et al. “HealthBench: Evaluating Large Language Models Towards Improved Human Health.” arXiv:2505.08775, May 2025. https://arxiv.org/abs/2505.08775
  7. Patwardhan, Tejal et al. “GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks.” arXiv:2510.04374, October 2025. https://arxiv.org/abs/2510.04374
  8. Whitfill, Parker, Cheryl Wu, Joel Becker, and Nate Rush. “Many SWE-bench-Passing PRs Would Not Be Merged into Main.” METR, March 10, 2026. https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/
  9. Baumann, Joachim et al. “SWE-chat: Coding Agent Interactions From Real Users in the Wild.” arXiv:2604.20779, April 2026 (v2 October 2026). https://arxiv.org/abs/2604.20779
  10. Lei, Fangyu et al. “Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows.” arXiv:2411.07763, November 2024. https://arxiv.org/abs/2411.07763
  11. Huang, Yiming et al. “DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models.” arXiv:2410.07331, October 2024. https://arxiv.org/abs/2410.07331
  12. Xie, Tianbao et al. “OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments.” arXiv:2404.07972, April 2024. https://arxiv.org/abs/2404.07972
  13. Zhou, Shuyan et al. “WebArena: A Realistic Web Environment for Building Autonomous Agents.” arXiv:2307.13854, July 2023. https://arxiv.org/abs/2307.13854
  14. Trivedi, Harsh et al. “AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents.” arXiv:2407.18901, July 2024. https://arxiv.org/abs/2407.18901
  15. Huang, Kung-Hsiang et al. “CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions.” arXiv:2505.18878, May 2025. https://arxiv.org/abs/2505.18878
  16. Jha, Saurabh et al. “ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks.” arXiv:2502.05352, February 2025. https://arxiv.org/abs/2502.05352
  17. Lu, Pengrui et al. “AlphaEval: Evaluating Agents in Production.” arXiv:2604.12162, April 2026. https://arxiv.org/abs/2604.12162
  18. Zhou, Jingbo et al. “OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding.” arXiv:2607.27155, July 2026 (v2 August 2026). https://arxiv.org/abs/2607.27155
  19. Jafari Meimandi, Kiana et al. “The Measurement Imbalance in Agentic AI Evaluation Undermines Industry Productivity Claims.” arXiv:2506.02064, June 2025 (v3 October 2025). https://arxiv.org/abs/2506.02064
  20. Gaonkar, Radhika. “TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation.” arXiv:2610.11678, October 2026. https://arxiv.org/abs/2610.11678
  21. Zhao, Enhan et al. “BizAgentBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business.” arXiv:2610.20424, October 2026. https://arxiv.org/abs/2610.20424
  22. Uesato, Jonathan et al. “Solving Math Word Problems with Process- and Outcome-Based Feedback.” arXiv:2211.14275, November 2022. https://arxiv.org/abs/2211.14275
  23. Huang, Jiaju et al. “Task Success Is Not Enough: Side-effect-Aware Evaluation of Tool-Using Language Model Agents.” arXiv:2610.34167, October 2026. https://arxiv.org/abs/2610.34167
  24. Tian, Yifang et al. “SRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability Agents.” arXiv:2609.33023, September 2026. https://arxiv.org/abs/2609.33023
  25. Du, Mingxuan et al. “DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents.” arXiv:2506.11763, June 2025. https://arxiv.org/abs/2506.11763
  26. Liu, Yujia et al. “Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift.” arXiv:2608.29543, August 2026. https://arxiv.org/abs/2608.29543
  27. Chen, Joy et al. “ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices.” arXiv:2608.26204, August 2026. https://arxiv.org/abs/2608.26204
  28. Li, Yunhan et al. “LegalCiteTrust: Benchmarking Citation Trustworthiness in Chinese Long-Form Legal Research Reports.” arXiv:2607.20872, July 2026. https://arxiv.org/abs/2607.20872
  29. Sun, Yubo et al. “HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research.” arXiv:2607.25151, July 2026. https://arxiv.org/abs/2607.25151
  30. Liu, Yumou et al. “DeliverBench: Benchmarking AI Agents on Delivering Realistic Multi-Stage Professional Projects.” arXiv:2611.03412, November 2026. https://arxiv.org/abs/2611.03412
  31. Roy, Soham et al. ""I Strongly Suspect This Website Is a Scam”: Benchmarking PII Leakage and Detection without Defense in Autonomous Web Agents.” arXiv:2606.00497, May 2026. https://arxiv.org/abs/2606.00497
  32. Meyer, Joshua et al. “VAmoS Bench: Voice Agent Simulation Bench.” arXiv:2607.27453, July 2026. https://arxiv.org/abs/2607.27453
  33. Qin, Yihua et al. “R2A-Sales: A Policy-Atom-Matched Benchmark for Rule-to-Action Gaps in Goal-Directed Sales Agents.” arXiv:2610.34185, October 2026. https://arxiv.org/abs/2610.34185
  34. Purwar, Anupam, Shashank Singh, and Kritika Srivastava. “Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight.” arXiv:2608.24314, August 2026 (v2). https://arxiv.org/abs/2608.24314
  35. Salimi, Ahmad et al. “IHBench: Evaluating Post-Interruption Recovery in Voice Agents with Structured Workflows.” arXiv:2606.19595, June 2026. https://arxiv.org/abs/2606.19595
  36. Ray, Soham et al. “τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains.” arXiv:2603.13686, March 2026. https://arxiv.org/abs/2603.13686
  37. Yu, Kai et al. “Does Pass Rate Tell the Whole Story? Evaluating Design Constraint Compliance in LLM-based Issue Resolution.” arXiv:2604.05955, April 2026. https://arxiv.org/abs/2604.05955
  38. Xia, Chunqiu Steven et al. “Do These Violent Delights Have Violent Ends? Measuring the Post-Merge Fate of Agentic Code.” arXiv:2607.09902, July 2026. https://arxiv.org/abs/2607.09902
  39. Zhao, Bingchen et al. “SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents.” arXiv:2605.21384, May 2026. https://arxiv.org/abs/2605.21384
  40. Mittal, Aditya et al. “Comparing Developer and LLM Biases in Code Evaluation.” arXiv:2603.24586, March 2026. https://arxiv.org/abs/2603.24586
  41. Steenhoek, Benjamin et al. “Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE.” arXiv:2412.14306, December 2024. https://arxiv.org/abs/2412.14306
  42. Drozdov, Katrina et al. “Legal Research Bench: Measuring End-to-End Reliability in Long-Horizon Legal Research Agents.” arXiv:2610.00609, September 2026. https://arxiv.org/abs/2610.00609
  43. Liu, Shuang et al. “ContractEval: Benchmarking LLMs for Clause-Level Legal Risk Identification in Commercial Contracts.” arXiv:2508.03080, August 2025. https://arxiv.org/abs/2508.03080
  44. Roy Choudhury, Manan et al. “Better Call CLAUSE: A Discrepancy Benchmark for Auditing LLMs Legal Reasoning Capabilities.” arXiv:2511.00340, November 2025. https://arxiv.org/abs/2511.00340
  45. Zhong, Joey et al. “DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity.” arXiv:2602.11685, February 2026. https://arxiv.org/abs/2602.11685
  46. Fan, Tianyu et al. “Understanding DeepResearch via Reports.” arXiv:2510.07861, October 2025. https://arxiv.org/abs/2510.07861
  47. Wang, Jiayu et al. “LiveResearchBench: A Live Benchmark for User-Centric Deep Research in the Wild.” arXiv:2510.14240, October 2025. https://arxiv.org/abs/2510.14240
  48. Li, Jinyang et al. “Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs.” arXiv:2305.03111, May 2023. https://arxiv.org/abs/2305.03111
  49. Dong, Mingwen et al. “PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries.” arXiv:2410.11076, October 2024. https://arxiv.org/abs/2410.11076
  50. Chen, Nan et al. “VisEval: A Benchmark for Data Visualization in the Era of Large Language Models.” arXiv:2407.00981, July 2024. https://arxiv.org/abs/2407.00981
  51. Lei, Fangyu et al. “DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle.” arXiv:2512.04324, December 2025. https://arxiv.org/abs/2512.04324
  52. Tien, Jeremy et al. “ROGUE: Evaluating Corrigibility Failures in Frontier Computer-Use Agents.” arXiv:2606.00341, May 2026. https://arxiv.org/abs/2606.00341
  53. Lù, Xing Han et al. “AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories.” arXiv:2504.08942, April 2025 (v2). https://arxiv.org/abs/2504.08942
  54. Su, Xuanbo et al. “Sell More, Play Less: Benchmarking LLM Realistic Selling Skill.” arXiv:2604.07054, April 2026. https://arxiv.org/abs/2604.07054
  55. Zhang, Qingyu et al. “AI-Salesman: Towards Reliable Large Language Model Driven Telemarketing.” arXiv:2511.12133, November 2025. https://arxiv.org/abs/2511.12133
  56. Fu, Andre et al. “Incident-Arena: Getting agents to the last nine of reliability.” arXiv:2610.00648, September 2026. https://arxiv.org/abs/2610.00648
  57. Chen, Yinfang et al. “STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds.” arXiv:2506.02009, May 2025. https://arxiv.org/abs/2506.02009
  58. Lu, Qisheng et al. “Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis.” arXiv:2608.21310, August 2026. https://arxiv.org/abs/2608.21310
  59. Mazeika, Mantas et al. “Remote Labor Index: Measuring AI Automation of Remote Work.” arXiv:2510.26787, October 2025. https://arxiv.org/abs/2510.26787
  60. Ali, Abbas Raza et al. “EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise.” arXiv:2609.21841, September 2026. https://arxiv.org/abs/2609.21841
  61. Mohammadi, Hadi. “trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories.” arXiv:2609.00038, August 2026 (v3 October 2026). https://arxiv.org/abs/2609.00038
  62. Anthropic. “Anthropic Economic Index report: Economic primitives.” Anthropic, January 15, 2026. https://www.anthropic.com/research/anthropic-economic-index-january-2026-report
  63. Cao, Hongliu. “Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error.” arXiv:2609.14400, September 2026. https://arxiv.org/abs/2609.14400
  64. Weng, Shihao, Yang Feng, and Xiaofei Xie. “Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges.” arXiv:2605.06161, May 2026. https://arxiv.org/abs/2605.06161
  65. Qin, Suliu, Lu Yin, and Xilu Wang. “Grounded Checklist Partial Credit for Agent Skill Trajectories.” arXiv:2608.27487, August 2026. https://arxiv.org/abs/2608.27487
  66. Zhao, Justin et al. “Jagged Judges: Epistemic Stability Under Perturbation, Pressure, and Persistence.” arXiv:2608.12645, August 2026 (v2). https://arxiv.org/abs/2608.12645
  67. Yegge, Steve. “The Counting House.” October 27, 2026. https://yegge.ai/essays/the-counting-house/
  68. Wen, Jiaxin et al. “Language Models Learn to Mislead Humans via RLHF.” arXiv:2409.12822, September 2024. https://arxiv.org/abs/2409.12822
  69. Sharma, Mrinank et al. “Towards Understanding Sycophancy in Language Models.” arXiv:2310.13548, October 2023. https://arxiv.org/abs/2310.13548
  70. Becker, Joel et al. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” arXiv:2507.09089, July 2025. https://arxiv.org/abs/2507.09089
  71. Mozannar, Hussein et al. “The RealHumanEval: Evaluating Large Language Models’ Abilities to Support Programmers.” arXiv:2404.02806, April 2024. https://arxiv.org/abs/2404.02806
  72. Lundh, Andreas et al. “Industry sponsorship and research outcome.” Cochrane Database of Systematic Reviews, 2017. https://doi.org/10.1002/14651858.MR000033.pub3
  73. Liu, Shang et al. “Incentivizing High-Quality Human Annotations with Golden Questions.” arXiv:2505.19134, May 2025. https://arxiv.org/abs/2505.19134
  74. Checco, Alessandro, Jo Bates, and Gianluca Demartini. “Adversarial Attacks on Crowdsourcing Quality Control.” Journal of Artificial Intelligence Research 67, 2020. https://doi.org/10.1613/jair.1.11332
  75. Anthropic. “Demystifying evals for AI agents.” Anthropic Engineering, 2026-01-09. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
  76. Dawid, A. P. and Skene, A. M. “Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm.” Journal of the Royal Statistical Society, Series C (Applied Statistics) 28(1):20–28, 1979. https://doi.org/10.2307/2346806
  77. Froger, Romain et al. “ARE: Scaling Up Agent Environments and Evaluations.” arXiv:2509.17158, September 2025 (v2 December 2025). https://arxiv.org/abs/2509.17158
  78. Starace, Giulio et al. “PaperBench: Evaluating AI’s Ability to Replicate AI Research.” arXiv:2504.01848, April 2025. https://arxiv.org/abs/2504.01848
  79. Zheng, Lianmin et al. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” arXiv:2306.05685, June 2023. https://arxiv.org/abs/2306.05685
  80. OpenAI. “Why SWE-bench Verified no longer measures frontier coding capabilities.” OpenAI, February 23, 2026. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
  81. Chen, Junjie. “When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark.” arXiv:2610.16067, October 2026. https://arxiv.org/abs/2610.16067
  82. Snow, Rion, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. “Cheap and Fast, But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks.” EMNLP 2008, pp. 254–263. https://aclanthology.org/D08-1027/
  83. Turney, Peter D. “Types of Cost in Inductive Concept Learning.” Workshop on Cost-Sensitive Learning, ICML 2000; arXiv:cs/0212034, posted December 2002. https://arxiv.org/abs/cs/0212034
  84. Zadrozny, Bianca and Charles Elkan. “Learning and Making Decisions When Costs and Probabilities are Both Unknown.” Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2001. https://cseweb.ucsd.edu/~elkan/kddbianca.pdf
  85. Kang, Kabir and Stephen Mussmann. “Instance-Level Costs for Nuanced Classifier Evaluation.” arXiv:2605.03135, May 2026 (v2 July 2026). https://arxiv.org/abs/2605.03135
  86. Keeney, Ralph L. and Howard Raiffa. Decisions with Multiple Objectives: Preferences and Value Tradeoffs. Wiley, 1976.
  87. Petersen, Ole. “Autoresearch at Listen.” Listen Labs blog, July 14, 2026. https://listenlabs.com/blog/autoresearch-at-listen
  88. Wahi, Vansh. “LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails.” arXiv:2609.02246, September 2026. https://arxiv.org/abs/2609.02246
  89. Gao, Leo, John Schulman and Jacob Hilton. “Scaling Laws for Reward Model Overoptimization.” arXiv:2210.10760, October 2022. https://arxiv.org/abs/2210.10760
  90. Altman, Eitan. Constrained Markov Decision Processes. Chapman and Hall/CRC, 1999.
  91. Miettinen, Kaisa. Nonlinear Multiobjective Optimization. Kluwer, 1999.
  92. Kapoor, Sayash et al. “AI Agents That Matter.” arXiv:2407.01502, July 2024. https://arxiv.org/abs/2407.01502
  93. Erol, Mehmet Hamza et al. “Cost-of-Pass: An Economic Framework for Evaluating Language Models.” arXiv:2504.13359, April 2025 (v2). https://arxiv.org/abs/2504.13359
  94. Cunningham, Tom, Manish Shetty, Vincent Cheng and Nate Rush. “Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT.” METR blog, July 21, 2026. https://metr.org/blog/2026-07-21-expenditure-horizon/
  95. Qian, Xingzhi et al. “ParetoSaddler: Pareto-Guided Optimization of Agent Harnesses for Quality and Cost.” arXiv:2611.02079, November 2026. https://arxiv.org/abs/2611.02079