Mahal Systems Overview › Benchmark catalog
Agent benchmark catalog
364 public agent benchmarks. In this method they are a source of grading designs, case templates, user simulators, sandbox designs and baseline numbers; the cases themselves come from your agent's own traffic.
These entries describe public benchmarks built and published by others; we don't host or redistribute their data. Licenses, terms and data handling are each benchmark's own, and some were assembled from real or scraped records that may include personal information. Check a benchmark's license and data card before you download or use it.
-
AgencyBench
Fits long-running coding or deliverable-building agents that take iterative feedback. Borrow the user-simulation agent that supplies feedback and the mixed rule, vision and LLM judges on deliverables.
general assistantcodingresearchtool calling
-
AgentAtlas
Fits any team deciding what to measure beyond pass/fail: the six control decisions and nine failure categories are a ready vocabulary for labeling an agent's trajectories.
general assistantsafetyplanning
-
AgentProcessBench
Fits teams building a step-level judge or process reward signal for a tool-using agent. Borrow the +1/0/-1 step labels with the error-propagation rule and the reported human agreement (89.1%).
tool callinggeneral assistantknowledge qa
-
AgentRx
Fits teams debugging a deployed agent across API workflows, incident management or web tasks. Borrow the constraint-synthesis-then-check pattern that gives an auditable violation log before the judge decides.
general assistantit opstool callingweb browsing
-
AgentThreatBench
Fits a quick smoke test for email or support agents mapped to the OWASP agentic top 10. Borrow the two independent binary scores per sample and the four outcome profiles (robust, vulnerable, overly cautious, failing).
safetysecuritytool callingemail calendarcustomer support
-
AMA-Bench
Fits long-running agents whose memory must cover their own tool calls and observations, not a human chat. Borrow the idea of mining real trajectories (from ALFWorld, WebArena, SWE-bench, GAIA, Spider 2.0) and writing expert questions over them.
memorygeneral assistantcodingweb browsing
-
APEX-Agents
Fits agents doing analyst work in files and office tools. Borrow the "world" seeds (31 workspaces, 5,182 files) and the binary-criteria rubrics judged per criterion.
financelegaldocumentsspreadsheetsgeneral assistant
-
Ask or Assume? (clarification-seeking in coding agents)
Fits coding agents and any agent that should ask before acting on vague requests. Borrow the recipe of stripping details from SWE-bench Verified issues and scoring both the resolve rate and how many questions were asked.
codinggeneral assistant
-
CI-Bench
Fits CI-repair or build-fixing agents; borrow the log-to-prompt pipeline and the re-execution in the exact failing container as the pass check.
devopscoding
-
ClawsBench
Fits productivity assistants wired to Workspace and Slack. Borrow the mock services with snapshot/restore and the unsafe-action-rate metric reported alongside success.
email calendardocumentsworkflow automationsafetygeneral assistant
-
CodeIPI
Fits coding agents that read issues and repos. Borrow the canary design: planted secrets and env vars for exfiltration, a unique marker for execution, and file hashes before and after for persistence.
safetysecuritycoding
-
CTI-REALM
Fits detection-engineering or security-operations agents. Borrow the weighted checkpoint scoring that credits CTI analysis, MITRE mapping, data exploration and query execution on the way to the final rule, and the F1 scoring of a detection rule against ground truth.
securityit opstool calling
-
DataClawEval
Fits data engineering agents that write jobs for several engines and must verify their own output. Borrow its deterministic rule-based graders and its score that blends artifact quality with process quality (exploration, efficiency, self-verification).
devopssqlcodingdata analysis
-
DeepPlanning
Fits booking or purchasing assistants with global budgets. Borrow the rule-based checkers (commonsense checkpoints, cart match) instead of LLM judges.
planningtravelshoppingtool calling
-
DSAEval
Fits full data science agents that produce notebooks and reports over several requests. Borrow the three-part judge rubric (reasoning, code, results) with a consistency flag that checks the narrative against what the code produced.
data analysisml engineeringscience
-
E-Commerce Bench
Fits agents that operate a business over a long horizon. Borrow the deterministic demand and negotiation kernels (the LLM only renders dialogue) and the multi-axis scoring with assets as the headline.
retailsalesplanningfinance
-
EnterpriseBench CoreCraft
Fits support agents built on MCP tools that must get every policy detail right. Borrow the all-or-nothing expert rubric scoring and the RL-environment framing.
customer supportretailtool callingworkflow automation
-
EnterpriseClawBench
Fits teams who have their own agent logs and want to turn them into evals. Borrow the construction pipeline (fixtures recovered, prompts rewritten, rubrics and rules generated) and the harness-plus-model reporting.
documentsdata analysisworkflow automationgeneral assistant
-
EnterpriseOps-Gym (NOWAI-Bench)
Fits ITSM, HR and customer-service agents that write to a system of record. Borrow the SQL verifiers on final database state (about 5 conditions per task, all must pass) and the one-container-per-domain MCP setup.
it opshrcustomer supportemail calendarworkflow automation
-
ERP-Bench (Anchor)
Fits ERP and back-office automation agents. Borrow the parametric task generator that yields a solver-certified optimal solution and a state-based verifier per task.
workflow automationmanufacturingfinanceplanning
-
ExploitBench
Fits advanced exploit-development or vulnerability-research agents. Borrow the 16 binary capability flags checked by deterministic oracles, which give graded progress instead of a single pass or fail.
securitycoding
-
Hack-Verifiable Terminal Bench (HVTB)
Fits coding and operations agents that run against tests, where reward hacking is a worry. Borrow the trick of planting a reference solution and tests in the container and using filesystem watchers to flag any read, giving a hack rate with no judge.
safetycodingit ops
-
JourneyBench
Fits contact-center agents that must follow a business flow drawn as a graph. Borrow the User Journey Coverage Score (zero unless the tool-call sequence matches exactly, then the fraction of correct parameters) and the graph representation of journeys used to generate scenarios.
customer supporttelecomfinanceshopping
-
LoCoMo-Plus
Fits assistants where remembered preferences or constraints must shape later answers even when the user does not repeat them. Borrow the cue-then-trigger case design and the correct/partial/wrong judge rubric.
memoryknowledge qageneral assistant
-
LongDA
Fits analytics agents whose hard part is reading documentation, such as codebooks or data dictionaries, before querying data. Borrow the two metrics: answer structure coverage and numeric match within a tolerance.
data analysisdocumentsresearch
-
Manager Coercion Benchmark (MCB)
Fits orchestrator agents that delegate to sub-agents; it checks whether the orchestrator turns coercive or fabricates success. Borrow the nine-rung escalation ladder recorded through a required tool argument so no judge is needed for the headline score.
multi agentsafety
-
Mem2ActBench
Fits tool-using assistants that should act on remembered user details rather than merely recall them. Borrow the reverse-generation trick of writing the memory chain first, then the task that depends on it.
memorytool callinggeneral assistant
-
MemoryArena
Fits agents whose sessions build on each other (a buyer, planner or researcher that returns to an unfinished job). Borrow the interdependent-subtask design where memory errors compound.
memoryshoppingtravelweb browsingplanning
-
MMDeepResearch-Bench
Fits research agents whose inputs include charts, screenshots or photos. Borrow the three separate scores for report quality, citation fidelity and text-to-image consistency, and the pass/fail gate on visual evidence.
researchweb browsingdocuments
-
MT-AgentRisk
Fits agents that take many turns with MCP tools. Borrow the recipe that turns a single-turn harmful task into an addition or decomposition attack sequence, and the complete/rejected/ failed judge labels.
safetytool callingweb browsingsqlcoding
-
OccuBench
Fits tool-using agents in niche occupations where no real sandbox exists. Borrow its idea of an LLM that plays the environment, and its fault-injection modes (explicit error, silent data degradation, mixed).
general assistanttool callingworkflow automation
-
OpenSanctions Pairssaturated
Fits compliance screening or know-your-customer agents that match entities against sanctions lists. Borrow its pairwise labeled format from 293 sources across 45 jurisdictions and its rule-based baseline (91.3% F1) as a floor.
data analysisfinancegovernmentsecurity
-
OSWorld 2.0
Fits long-running desktop agents; the successor to OSWorld-Verified with headroom. Borrow binary completion plus partial-credit scoring and the safety-sensitive action audit.
computer usedocumentsdata analysisworkflow automation
-
PersistBench
Fits assistants with persistent user memory. Borrow its two failure classes, cross-domain leakage and memory-induced sycophancy, as test categories.
memorysafetygeneral assistant
-
ProjDevBench
Fits agents that build whole programs end to end; borrow the 80/20 split between execution score and code review score and the rule-based checks for forbidden shortcuts.
coding
-
SalesLLM
Fits outbound or inbound sales chat agents with no backend actions. Borrow CustomerLM, a fine-tuned customer simulator that resists role inversion, and the two-part score: an LLM judge for sales-process progress plus a BERT classifier for end-of-dialogue buying intent.
salesfinanceretail
-
SupChain-Bench
Fits order-management, logistics operations and back-office agents that chain many read-only API calls under a procedure. Borrow the oracle-script ground truth (expected tool outputs per question), the entity-level precision and recall scoring, and the three difficulty levels by step count.
manufacturingworkflow automationtool callingknowledge qa
-
SWE-PRBench
Fits code-review agents that choose how much context to load; its finding that more context lowered every model's score is directly useful for prompt design.
code review
-
SycoBench-600
Fits agents that must hold a correct answer under user pressure but still take valid corrections, such as support or advisory bots. Borrow the correction-selectivity metric (update rate minus wrong-flip rate) and the matched correct-suggestion turn.
safetyknowledge qa
-
TelcoAgent-Bench
Fits network-operations or technical-support agents in telecom rather than consumer customer support. Borrow the metric suite (intent recognition, ordered tool execution, resolution correctness, stability across variants) and the blueprint-with-sampled-KPIs method for generating variants.
telecomit opstool calling
-
TeleCom-Bench
Fits network-operations agents that triage faults and run optimization workflows. Borrow its six-step decomposition (intent, entities, tool call, event check, root cause, solution) and its mix of macro-F1, exact match and a 5-point judge with reported rater agreement.
telecomit opstool callingknowledge qa
-
UNDERWRITE
Fits insurance and lending agents that must collect information before deciding. Borrow the imperfect simulated user and the pass^k reliability drop as a reported metric.
financecustomer supporttool calling
-
Vibe Code Bench
Fits app-builder agents; borrow the idea of grading a deployed app by having a browser agent walk 964 scripted workflows instead of unit tests.
coding
-
WeaveBench
Fits coding agents that also need a GUI. Borrow the trajectory-aware judge that flags shortcuts such as faked screenshots or hard-coded results.
computer usecodingworkflow automation
-
WebArena-Infinity
Fits teams that need many fresh, uncontaminated browser tasks; borrow the pipeline that generates an app, its tasks and verifiers with coding agents and audits them with browser agents.
web browsingworkflow automation
-
World of Workflows (WoW-bench)
Fits agents that write to systems with hidden business rules (ITSM, ERP). Borrow the table audit idea: grade on database state changes, including cascading ones the agent never saw.
it opsworkflow automationplanning
-
τ³-bench
Fits support agents that must answer from a knowledge base as well as act on a backend, and voice agents. Borrow the banking_knowledge domain pattern (retrieval graded against gold read calls) and the voice evaluation setup with simulated callers.
customer supportfinanceknowledge qatelecomtool calling
-
τᶜ-Bench (TASTE)
Fits teams whose agent already does well on τ²-bench and who need harder cases on the same tools. Borrow the method: sample valid tool sequences, cluster them for coverage, then write the task around the sequence and tune its difficulty.
customer supporttelecomretailtraveltool calling
-
ACEBench
Fits assistants that must refuse or ask back when a request is ambiguous or unservable, not just call tools; borrow its Normal / Special / Agent split and its rule-based scoring with end-state and process accuracy for the multi-turn subset.
tool calling
-
AgentArch
Fits teams choosing an agent architecture for enterprise work. Borrow the grid of 18 configurations (orchestration, memory, thinking, tool style) as an ablation plan.
workflow automationtool callingit ops
-
AgentAuditor / ASSEBench
Fits teams building an LLM judge for agent safety rather than testing the agent itself. Borrow the human-labeled records with ambiguity flags and the strict/lenient split to calibrate a judge's F1 before trusting it.
safetysecuritytool calling
-
Agentic Misalignment
Fits any agent given standing access to a company inbox or documents plus a long-running goal, such as an executive assistant or operations agent. Borrow the way one scenario is crossed with goal type, goal content and urgency to build a condition grid, and the classifier prompt that labels a transcript harmful or not.
safetyemail calendargeneral assistant
-
AIM-Bench
Fits inventory or replenishment agents that act in a loop. Borrow the outcome metrics (holding and stockout cost, stockout rate, turnover), the distance-from-optimal-order per-step metric, and the bias probes (anchoring, demand chasing, bullwhip) as trajectory checks.
manufacturingretailplanning
-
AIOpsLab
Fits incident-response agents; borrow the four-stage taxonomy (detection, localization, analysis, mitigation) and the problem = app + fault + workload + evaluator structure.
it opsdevops
-
AndroidBuildBench
Fits build-and-dependency repair agents; its "tool bridging" result (domain tools beat a raw shell) is a design lesson for any devops agent.
devopscodingmobile
-
AssetOpsBench
Fits maintenance, condition-monitoring and asset-management agents in plants or utilities. Borrow its MCP-server sandbox (CouchDB work orders with Maximo field names, a read-only mode) and its comparison of tool-as-agent versus plan-executor designs.
manufacturingit opstool callingmulti agentdata analysis
-
AstaBench
Fits scientific research-assistant agents that combine literature search, code execution and data analysis. Borrow its reproducible search environment with date cutoffs, its cost-per-task reporting next to accuracy, and its four-category macro average.
scienceresearchcodingdata analysisplanning
-
b3 (Backbone Breaker Benchmark)
Fits choosing a backbone model for an agent before building it. Borrow the threat snapshot idea (freeze one step of the agent where the model sees an attack) and the three defense levels per snapshot.
safetysecuritytool callinggeneral assistant
-
BEARCUBS
Fits multimodal browsing agents; borrow the rule that each question has one short, unambiguous answer plus a human-validated trajectory.
web browsingresearchknowledge qa
-
BioProBench
Fits lab-assistant or protocol-writing agents that must respect procedural order, quantities and safety steps. Borrow the five task shapes (QA, ordering, error correction, generation, reasoning) and the step recall and Kendall tau metrics for ordered procedures.
scienceknowledge qaplanning
-
BIRD-Critic (SWE-SQL)
Fits SQL support or database debugging agents that take a user complaint and a broken query and must return a working one. Borrow its test-case grading on the live database and its variants (Open 570 multi-dialect, PostgreSQL 530, Flash 200, BigQuery 200).
sqlcodingit ops
-
BIRD-Interact
Fits conversational database assistants that must handle vague requests, clarification and write operations. Borrow the function-driven user simulator, the two modes (scripted c-Interact and open a-Interact) and the per-task executable test cases.
sqldata analysiscustomer support
-
BixBench
Fits bioinformatics or data-analysis agents that work inside a notebook against real data. Borrow the "capsule" pattern (a published notebook plus its data turned into open questions) and the Docker notebook sandbox.
sciencedata analysiscoding
-
BizFinBench
Fits retail-investor assistants and finance chat agents. Borrow the IteraJudge idea (an iterated judge that refines its decision boundary) and the nine-category task split when designing a judge for objective finance answers.
financeknowledge qadata analysistool calling
-
BrowseComp
Fits agents whose job is to dig up obscure facts on the live web and return a short answer. Borrow the pattern of building questions backward from a short verifiable answer, and the accuracy plus calibration reporting.
web browsingresearchknowledge qa
-
BrowseComp-Plus
Fits search agents that run over an internal document store rather than the open web. Borrow the fixed corpus with human-verified positives and mined hard negatives, and the separate retrieval metrics (nDCG, recall) next to answer accuracy.
web browsingresearchknowledge qa
-
BrowseComp-ZH
Fits agents that must search Chinese-language sites, where English benchmarks say little. Borrow the reverse-engineered question design and the encrypted dataset with a canary token.
web browsingresearchknowledge qa
-
BrowserArena
Fits teams that want pairwise human preference on live browsing rather than fixed tasks. Borrow the step-level human feedback and the captcha / pop-up / URL failure taxonomy.
web browsinggeneral assistant
-
CAIBenchfrozen
Fits teams building both attack and defense agents who want one place to measure knowledge, CTF and range tasks. Borrow the split into five categories and the attack-defense setup where two agents face each other.
securityknowledge qamulti agent
-
ComplexFuncBench
Fits travel or booking agents and any agent with long, constrained parameter lists; borrow ComplexEval's layered matching (format check, exact match, then equal-response match, then a GPT-4o tie-break) for grading calls that have more than one right form.
tool callingtravel
-
ContextCRBench
Fits teams building a review bot with issue context; borrow the three granularities (hunk, line, comment) as a taxonomy and the context-enrichment fields per record.
code review
-
CORGI
Fits business-analyst agents whose users ask why and what-next questions, not only lookups. Borrow its four-level question taxonomy (descriptive, explanatory, predictive, recommendational) and its success execution rate metric.
sqldata analysis
-
CRMArena-Pro
Fits sales-assist and service agents that work inside a CRM, in both single-shot and conversational modes. Borrow the 19-task skill taxonomy (workflow execution, policy compliance, information retrieval, textual reasoning), the paired single-turn versus multi-turn scoring, and the confidentiality-awareness probes.
crmsalescustomer supportknowledge qatool calling
-
CVE-Bench
Fits penetration-testing or web-security agents. Borrow the eight concrete attack outcomes checked by graders (not just a flag), and the zero-day versus one-day prompt variants.
securityweb browsing
-
CyberGym
Fits code-security or bug-finding agents that work on real repositories. Borrow the execution oracle that runs the agent's input against a vulnerable and a fixed build, and the four difficulty levels that vary how much context is given.
securitycoding
-
DABstep
Fits finance or operations analytics agents that answer questions from files plus policy documents. Borrow its factoid grading without a judge model (numeric tolerance, fuzzy strings, element-wise lists) and its approach of making synthetic data with real-shaped business context.
data analysisfinancecoding
-
DAComp
Fits teams whose agent spans both pipeline building and analysis, since it keeps the two separate and shows they need different skills. Borrow its split of execution-based grading for engineering tasks and rubric-guided judge grading for analysis tasks.
sqldata analysisdevops
-
DataSciBench
Fits agents that take vague analysis requests rather than crisp questions. Borrow the Task-Function-Code idea: each prompt is tied to named aggregate functions and programmatic rules that score the execution result, plus a semi-automatic way to build ground truth.
data analysiscodingscience
-
Deep Research Bench (FutureSearch)
Fits research agents that must stay reproducible as the web changes. Borrow the RetroSearch idea of freezing the pages per task so human-worked answers stay valid, and the trace checks for hallucination, tool use and forgetting.
researchweb browsing
-
DeepResearch Bench
Fits report-writing research agents. Borrow RACE (task-specific weighted criteria scored against a reference report) and FACT (extract claim-URL pairs and check whether the cited page supports each claim).
researchweb browsingdocuments
-
DeepResearchGym
Fits teams who need reproducible search for a research agent. Borrow the stable search sandbox over public corpora and the three judge dimensions: alignment with the information need, retrieval faithfulness, and report quality.
researchweb browsing
-
DeepScholar-Bench
Fits literature-review or scientific writing agents. Borrow the live-refresh pipeline that mines new arXiv papers for queries with human-written reference text, and the split metrics for organization, nugget coverage, reference coverage and citation precision.
researchscienceweb browsing
-
DeepShop
Fits shopping research assistants. Borrow the fine-grained scoring of attributes, filters and sort order plus the easy / medium / hard query evolution.
shoppingweb browsingresearch
-
DialogTool
Fits phone-assistant agents where API state (an open cart, a running timer) persists across turns. Its three-stage life cycle (create, use, respond) is a useful way to split test cases.
mobilegeneral assistanttool calling
-
DoomArena
Fits teams that already run tau-bench or a browser benchmark and want to add threat models without rewriting tasks. Borrow the separation of attack, attackable component and success criterion from the host environment.
safetysecurityweb browsingtool callingcustomer support
-
Dr. Bench
Fits report-writing agents that need more than a single judge score. Borrow the reference bundle per task: query-specific rubrics, general report rubrics, trusted source links, and focus-anchor versus focus-deviation keywords.
researchweb browsingdocuments
-
ECom-Bench
Fits e-commerce after-sales support agents. Borrow the persona files (consumption habits, personality) that drive the simulated shopper, and the three-part rule-based reward (end database state, required lookup tools called, required keywords in the reply) on top of the τ-bench harness.
customer supportshoppingretailtool calling
-
EduBench
Fits education assistants that serve both students and teachers. Borrow its nine-scenario taxonomy and the 12-indicator scoring sheet (scenario fit, factual accuracy, pedagogy), with human raters used to check the model grader.
educationknowledge qadocuments
-
ELT-Bench
Fits data engineering agents that build or maintain extract-load-transform pipelines. Borrow its two-stage grading that compares produced warehouse tables to ground-truth CSVs and its containerized source systems (835 source tables, 203 data models).
devopssqlworkflow automationcoding
-
EnterpriseBench
Fits internal assistants that must respect who can see which data across departments. Borrow its generation pipeline that builds consistent synthetic company data from org metadata.
hrfinancecodingworkflow automationknowledge qa
-
Evo-Memory
Fits agents that are expected to get better over a deployment by learning from their own past runs. Borrow the idea of turning a fixed test set into a sequential stream and measuring gain over time.
memoryknowledge qageneral assistant
-
ExCyTIn-Bench
Fits security operations or log-investigation agents that query a SIEM. Borrow the way questions are generated from an investigation graph so each has an explainable answer, and the controlled tenant that yields realistic logs without customer data.
securitysqlit ops
-
FailSafeQA
Fits document question answering agents in finance or any field where users upload the wrong file or type sloppy questions. Borrow its seven-condition design (clean query, three query perturbations, OCR noise, missing context, irrelevant context) and its robustness score that takes the minimum across conditions.
financedocumentssafety
-
FEA-Bench
Fits incremental-development agents in Python; the task format (issue plus new component signatures plus tests) is a usable template for feature cases.
coding
-
FHIR-AgentBench
Fits read-only EHR question-answering agents that talk to a FHIR server. Borrow the two-stage scoring (resource retrieval recall and precision, then answer correctness) and the single-turn versus multi-turn and text versus code agent variants.
healthcaretool callingknowledge qa
-
FinAgentBench
Fits retrieval-augmented filing agents where the retrieval step is the weak point. Borrow the two-stage split (document-type ranking then chunk ranking) and the ten query categories to diagnose where retrieval fails before judging final answers.
financedocumentsresearch
-
Finance Agent Benchmark (Vals AI)
Fits research agents that answer analyst questions by searching the web and SEC filings. Borrow the nine-category task taxonomy and the per-fact rubric grading (separate correctness and contradiction checks per answer) rather than one overall judge call.
financeresearchweb browsingtool callingdocuments
-
FinGAIA
Fits general finance research agents in the GAIA style (web, files, code). Borrow the three scenario depths (operational analytics, asset decision, strategic risk) and the capability breakdown per task to size a case set.
financeweb browsingtool callingresearchdocuments
-
FinSearchComp
Fits finance research or market-data agents that must search and then compute. Borrow the three-tier task split (time-sensitive fetch, simple lookup, complex investigation) and the judge prompts with explicit numeric tolerance bands shipped with each question.
financeresearchweb browsingtool calling
-
FormFactory
Fits data-entry and onboarding agents. Borrow the atomic (one field) versus episodic (whole form) scoring split.
web browsingworkflow automationdocuments
-
FrontierCS
Fits agents that optimize rather than just pass; borrow the continuous scoring curve (60 points for matching the reference, log bonus for beating it, capped at 100).
codingresearch
-
FrontierScience
Fits science reasoning assistants where you want a hard, unsaturated reference number. Borrow its research-track rubrics that give partial credit for progress through a task rather than only the final answer.
scienceknowledge qaresearch
-
Gaia2 (Meta ARE)
Fits personal and workplace assistants that must act on time and cope with events arriving during a task. Borrow the write-action verifier against oracle events and the capability splits (execution, search, ambiguity, adaptability, time, noise, agent-to-agent).
general assistantemail calendartool callingmulti agentplanning
-
GDM Dangerous Capabilities: Stealth
Fits agents with write access to their own configuration, logs or monitoring, where quiet rule breaking would matter. Borrow the scenarios that put the agent near its own oversight and the use of realistic names so the model does not notice it is being tested.
safetyit opstool calling
-
GDPval
Fits document-producing assistants in knowledge-work roles. Borrow the occupation-by-task taxonomy (44 occupations, 9 sectors) and the weighted rubrics written by professionals.
documentsspreadsheetsfinancelegalgeneral assistant
-
HCAST
Fits teams who want to express agent ability as "how long a human would take". Borrow the human-time calibration (563 baselines) and the algorithmic scorer per task.
codingml engineeringsecuritygeneral assistant
-
HealthBench
Fits patient-facing or clinician-facing chat assistants with no tools. Borrow the per-conversation weighted rubric criteria (48,562 written by 262 physicians), the seven themes and behavior axes, and the Consensus and Hard subsets.
healthcareknowledge qasafety
-
Humanity's Last Exam
Fits general assistants where raw expert knowledge of the underlying model matters. Borrow the short-answer judge prompt and the calibration-error reporting next to accuracy.
general assistantknowledge qa
-
IFEvalCode
Fits coding assistants that must honor house style or constraints; borrow the split scoring of correctness versus instruction-following as two separate numbers.
coding
-
InnovatorBench
Fits ML-engineering or research agents that run multi-hour training experiments. Borrow the ResearchGym pattern (asynchronous monitoring, snapshots, distributed execution) and the task-specific evaluation scripts that score a runnable artifact on a held-out metric.
ml engineeringresearchcodingscience
-
InstrumentalEval
Fits teams giving an agent an open-ended goal such as make money or grow usage, who want a cheap screen for power-seeking or self-preservation talk. Borrow the six behavior categories as a taxonomy for red-team cases.
safetygeneral assistant
-
IntellAgent
Fits any policy-bound chat agent whose team wants to generate their own test cases from a prompt or policy document. Borrow the policy graph (policies as nodes, co-occurrence and difficulty as edges) used to sample multi-policy scenarios, and the critique step that maps failures back to specific policies.
customer supporttraveleducationtool calling
-
ITBench
The closest public match for SRE or on-call agents; borrow the push-button Kubernetes incident scenarios and the resolve-rate reporting by domain.
it opsdevopssecurity
-
KernelBench
Fits performance-engineering agents; borrow fast_p, which counts a task only if it is both correct and at least p times faster, as a two-part pass rule.
ml engineeringcoding
-
KramaBench
Fits analytics agents that work over many raw files rather than one tidy database. Borrow its sub-task decomposition (633 sub-tasks under 104 tasks), its mix of exact metrics, generated unit tests and rubric partial credit, and its six domain split.
data analysissciencecodingresearch
-
LiveCodeBench Pro
Fits teams that need a hard, uncontaminated algorithmic test with expert error labels; borrow the Olympiad-medalist failure taxonomy for grading reasoning errors.
coding
-
LiveDRBench
Fits research agents whose output should be a set of findings rather than prose. Borrow the claim-set intermediate representation graded by precision, recall and F1 per category.
researchweb browsingscience
-
LiveMCPBench
Fits agents that must find the right tool in a large catalog before using it; the paper attributes about half of failures to retrieval. Borrow its Dockerized key-free tool suite and its key-point rubric judged by an LLM against live outcomes.
tool callinggeneral assistantresearchtravelshopping
-
LiveSQLBench (BIRD-SQL Pro)
Fits text-to-SQL agents on PostgreSQL that must also do writes, and teams that worry about benchmark contamination since databases are added over time. Borrow its soft execution accuracy for SELECT tasks and custom test cases for management tasks.
sqldata analysis
-
LoCoBench-Agent
Fits agents that must navigate large repositories; borrow its efficiency metrics such as files read versus files modified and read-before-write checks.
coding
-
LogiOR (ORThought)
Fits planning or operations-research copilots that turn a logistics request into a solvable model. Borrow the case format (problem text, reference formulation, reference solver code, optimal value, size and type metadata) and grading on the optimal value.
planningdata analysiscodingmanufacturing
-
Lost in Conversation
Fits any chat agent whose users reveal requirements gradually. Borrow the sharded instruction format, the simulator that reveals one shard per turn, and the split of results into aptitude (90th percentile) and unreliability (90th minus 10th).
general assistantcodingsqltool callingdocuments
-
macOSWorld
Fits agents that must run on Mac desktops or in several languages. Borrow the multilingual instruction and interface variants and the deception subset.
computer usesafety
-
MASK
Fits agents whose operator prompt could push them to shade the truth, such as sales or support bots with a persona. Borrow the split between honesty and accuracy, and the belief-elicitation step that measures what the model thinks before applying pressure.
safetyknowledge qageneral assistant
-
MAST (Why Do Multi-Agent LLM Systems Fail)
Fits teams running orchestrator-plus-worker systems who need a failure taxonomy for trace review. Borrow the 14 modes in three groups and the released LLM annotator with its human-agreement study.
multi agentgeneral assistant
-
MathTutorBench
Fits math tutoring agents that must teach rather than just answer. Borrow the three-skill split (math expertise, student understanding, pedagogical ability) and the small trained reward model that scores a teacher turn by win rate against the real teacher's turn.
educationknowledge qa
-
MCP-Atlas
Fits general assistant agents that answer questions by orchestrating many MCP tools. Borrow its claim-level rubric (score the final answer against atomic factual claims, so different valid tool paths still pass) and its 11-category failure taxonomy that separates tool-call errors from reasoning errors.
tool callingresearchgeneral assistantknowledge qa
-
MCP-Bench
Fits agents that must discover and chain tools across several MCP servers from an underspecified request. Borrow its split of rule-based checks (tool name valid, schema compliance, call succeeded) from LLM-judged completion and planning scores, and its "description fuzzing" trick for writing tasks that do not name the tool.
tool callingresearchfinancetravelscience
-
MCP-RADARfrozen
Fits teams that care about efficiency (rounds, tokens, time) as well as correctness in tool use, and that want mock servers for email and calendar with a controlled local database. Borrow its accuracy-versus-efficiency reporting and its mock-server design.
tool callingemail calendarit opsknowledge qa
-
MCP-SafetyBench
Fits agents that connect to third-party MCP servers. Borrow the 20-type attack taxonomy across server, host and user sides, and the paired task evaluator plus attack detector that reports task success rate and attack success rate together.
safetysecuritytool callingweb browsingfinance
-
MCP-Universe
Fits agents that drive real third-party services (maps, GitHub, browser, market data) over long tool-calling sessions. Borrow its evaluator pattern (extract a value from the output, compare with an operator) and its "dynamic evaluators" that fetch live ground truth for time-sensitive tasks instead of freezing an answer.
tool callingweb browsingfinancedevopsresearch
-
MCPMark
Fits agents that write to Notion, GitHub or databases through MCP. Borrow the per-task verify.py against a curated initial state and the pass^k reliability reporting.
tool callingworkflow automationdocumentssqlcoding
-
MCPSecBench
Fits teams shipping an MCP host or client, not just a model. Borrow the 17 attack types over four surfaces (client, protocol, server, host) and the success/refusal/execution-error outcome split.
safetysecuritytool calling
-
MCPToolBench++
Fits agents whose main risk is malformed or wrong tool calls rather than long planning. Borrow its pairing of static AST accuracy with live Pass@k execution, which shows how much a model's "correct-looking" calls actually fail against real servers.
tool callingweb browsingshoppingfinancegeneral assistant
-
MCPTox
Fits agents that load tools from external MCP registries. Borrow the three poisoning templates and the 10 risk categories to build a tool-description screening test.
safetysecuritytool calling
-
MedAgentBench
Fits agents that read and write an electronic health record through standard APIs (chart review, order entry, documentation helpers). Borrow the Docker FHIR sandbox, the 10 task categories written by physicians, and the rule-based checks on POST payloads.
healthcaretool callingworkflow automation
-
MedAgentGym
Fits code-writing agents for clinical data analysis and biomedical research support. Borrow the Docker sandbox with request-info, code and debug actions, the 12-source task taxonomy, and the split between exact-match coding tasks and accuracy-scored ML tasks.
healthcarecodingdata analysissql
-
MedHELM
Fits any medical assistant whose team needs a map of clinical task categories. Borrow the clinician-validated taxonomy (5 categories, 22 subcategories, 121 tasks), the LLM-jury scoring validated against clinician ratings, and the public, gated and private access tiers.
healthcaredocumentsknowledge qaresearch
-
MemBench
Fits personal assistants that build a profile of a user over time. Borrow the split into factual versus reflective memory and participation versus observation, and the memory capacity and read/write time metrics.
memorygeneral assistant
-
MemoryAgentBench
Fits any agent with an external memory store; the four competencies (retrieval, test-time learning, long-range understanding, conflict resolution) are a usable taxonomy and the repo already wires in Letta, Cognee and HippoRAG. This is the closest match to the seed name "Agent Memory Benchmark".
memoryknowledge qageneral assistant
-
MEMTRACK
Fits engineering or project assistants that watch several work tools over time and must keep an accurate picture of what is true now. Borrow the time-ordered multi-platform timeline with planted conflicts and the correctness/efficiency/redundancy split.
memoryit opsdevopscoding
-
Mind2Web 2
Fits research assistants whose answers depend on current web state. Borrow Agent-as-a-Judge: a per-task judge script built from a tree rubric that opens the cited pages and checks both correctness and attribution.
web browsingresearch
-
MLR-Bench
Fits end-to-end research-writing agents; borrow the stage-by-stage rubric judge and the finding that about 80 percent of coding-agent results were fabricated as a check to add.
ml engineeringresearch
-
MLRC-Bench
Fits research-automation agents; borrow "fraction of the gap between baseline and top human closed" as a metric that avoids an LLM judge.
ml engineeringresearch
-
MobileWorld
Fits mobile agents in shopping and workplace apps. Borrow the backend-database checks for state and the user-interaction tasks.
mobileshoppingworkflow automationtool calling
-
MTRAG
Fits retrieval-augmented support or helpdesk assistants that hold a conversation over a knowledge base. Borrow its conversation-plus-corpus case format, its unanswerable and non-standalone question labels, and its separate retrieval and generation scoring.
knowledge qacustomer supportfinancegovernment
-
Multi-IaC-Bench
Fits agents that edit infrastructure templates rather than write them fresh; the (initial template, change request, target template) triplet is the case format to copy.
devopsit ops
-
Multi-Mission Tool Bench
Fits assistants whose users pile on related or changing requests in one session. The dynamic decision-tree grading that accepts several valid paths and scores path optimality is the part to borrow.
tool callinggeneral assistant
-
Multi-SWE-bench
The largest multi-language issue set; fits coding agents that serve polyglot codebases. Borrow the 68-annotator screening rubric and the mini (400) and flash (300) subsets for cost.
coding
-
MultiAgentBench (MARBLE)
Fits teams running several cooperating agents (planner plus workers, or peer agents). Borrow the milestone-based KPI where an LLM detector marks which sub-goals were hit, plus the separate coordination score.
multi agentresearchcodinggames
-
Online-Mind2Web
Fits general browser agents on public consumer sites. Borrow WebJudge, a screenshot-based LLM judge with released human labels to calibrate against.
web browsingshoppingtravelgeneral assistant
-
OpenAgentSafety
Fits internal workplace or devops agents that act on real services. Borrow the Docker company sandbox from TheAgentCompany, the rule-based final-state checker paired with a four-label trajectory judge, and the NPC users that push unsafe requests.
safetyworkflow automationcodingweb browsingtool calling
-
OS-Harm
Fits computer-use agents that drive a real desktop. Borrow the three-way split (user misuse, injection, model misbehavior) and the judge that scores accuracy and safety together with published human agreement.
safetycomputer useemail calendarcoding
-
OSUniverse
Fits teams that want a difficulty ladder for desktop agents. Borrow the five tiers (paper to gold) with step budgets and the checklist-based Gemini validator.
computer usegames
-
OSWorld-Human
Fits any desktop agent team that must report latency or cost, not just success. Borrow the weighted efficiency score that combines success with steps relative to a human.
computer use
-
OSWorld-MCP
Fits agents that mix GUI control with API or MCP tools. Borrow the tool-invocation-rate and average-completion-steps metrics alongside accuracy.
computer usetool callingdocuments
-
OSWorld-Verifiedsaturated
Fits general desktop agents on office and OS tasks; the current reference number for computer-use models. Borrow the parallel AWS runner that finishes a run within an hour.
computer usedocumentsspreadsheetsit ops
-
PaperArena
Fits agents that answer questions over a private document corpus with a tool set (parse, retrieve, compute). Borrow the modular tool platform and the question pipeline with obfuscation and manual review over a fixed corpus of 100 papers.
researchsciencetool calling
-
PaperBench
Fits research-replication agents; borrow the rubric tree (8,316 leaf requirements) and the Code-Dev variant that grades code only and skips the GPU reproduction step.
ml engineeringresearch
-
PatchEval
Fits vulnerability-repair agents in web languages; borrow the paired security plus functionality test as the pass rule.
securitycoding
-
REAL
Fits consumer browser agents that touch money or bookings. Borrow the replica-site sandbox with state checks for actions and rubric judging for lookups.
web browsingshoppingtravelemail calendar
-
REAL (Real Estate Agent LLM Evaluation)
Fits property-listing chat or brokerage assistants. Borrow the hallucination probes built from fictional block, area and road names judged by premise-response conflict, and the pipeline that filters names and phone numbers out of real brokerage data.
knowledge qacustomer supportsales
-
RealHiTBench
Fits assistants that read reports or financial tables with multi-level headers. Borrow the structure taxonomy (hierarchical headers, nested sub-tables, multi-table joins, decorative elements) and the practice of testing the same table in several input formats.
data analysisknowledge qadocuments
-
REALM-Bench
Fits planning and scheduling agents in operations or logistics. Borrow the six metrics (goal satisfaction, makespan, coordination, constraints, resource use, replanning).
planningmulti agentmanufacturing
-
Remote Labor Index (RLI)
Fits teams who want an economic value baseline for end-to-end work. Borrow the 3-point expert rating with majority vote and the Bradley-Terry pairing against a human reference.
general assistantdocumentsdata analysiscoding
-
ResearchQA
Fits assistants that answer domain research questions at scale. Borrow the method of mining questions and yes/no rubric items from survey articles so a large rubric set exists without expert annotators.
researchscienceknowledge qa
-
ResearchRubrics
Fits any report-writing agent that needs rubric grading. Borrow the expert-written weighted rubrics (20-43 criteria per prompt, with negative-weight criteria) and the three-level satisfied / partial / not-satisfied verdict scheme with multiple judge models.
researchweb browsingdocuments
-
SafeArena
Fits web agents that post, buy or publish on behalf of users. Borrow the safe/harmful task pairs and the four-level risk scale (refuse at once, refuse later, attempt and fail, complete) judged on the trajectory.
safetyweb browsingshopping
-
ScreenSpot-Pro (and ScreenSpot)
Fits any screenshot-driven agent whose failures are mis-clicks; use it to test the grounding model alone. Borrow the instruction-plus-bounding-box case format.
computer use
-
SCUBA
Fits CRM agents working in Salesforce. Borrow the persona-based task sourcing from user interviews and milestone scoring against org snapshots.
crmsalescustomer supportcomputer useweb browsing
-
SealQA
Fits search-augmented assistants that must not be fooled by bad search hits. Borrow the idea of selecting questions where search returns conflicting or unhelpful results, and the LongSeal needle-in-a-haystack variant for retrieval-heavy agents.
web browsingknowledge qa
-
SEC-bench
Fits security-fix agents; borrow the automated pipeline that reproduces a CVE in a container and the two-sided check (exploit no longer fires, build still passes).
securitycoding
-
SHADE-Arena
Fits agents that run many tool calls on a user's behalf with money or data at stake, and teams that want an LLM monitor on their traces. Borrow the main-task plus side-task pairing, the 0-100 suspicion score from a monitor that cannot see the system prompt, and the mock-tool environment design.
safetytool callinggeneral assistantfinanceemail calendar
-
ShoppingBench
Fits shopping assistants with budget or promotion logic. Borrow the intent-grounded instruction generation from sampled products.
shoppingretail
-
SOP-Bench
Fits agents that must execute a company procedure exactly. Borrow the metric split (execution completion rate times conditional success equals task success) and tool accuracy.
workflow automationcustomer supporthealthcarefinancemanufacturing
-
SOPBench (customer service)
Fits support agents whose work is defined by written procedures with preconditions (check this, then do that). Borrow the idea of encoding each procedure as code and using it as the oracle verifier for outcome, step and trajectory compliance, so no LLM judge is needed.
customer supportfinancehealthcaregovernmenttool calling
-
SWE-bench Multilingual
Fits coding agents that must work outside Python; 42 repos across 9 languages including redis, terraform, lucene and tokio. Use it to check that your harness is not Python-specific.
coding
-
SWE-bench Pro
Fits teams whose coding agent works on larger changes than SWE-bench; the held-out commercial split is the pattern to copy if you want a private test set that nobody can train on.
coding
-
SWE-Dev
Fits feature-implementation agents rather than bug fixers. Borrow the Easy/Hard test split and the 14,000-instance training set if you train your own model.
coding
-
SWE-EVO
Fits agents asked to do long multi-step work rather than one bug; borrow the partial-credit Fix Rate metric and the release-note-to-task construction.
coding
-
SWE-Lancer
Fits product-engineering agents on a large JavaScript/TypeScript app, and manager-style agents that pick between proposals. Borrow dollar-weighted scoring and Playwright end-to-end tests.
codingcode review
-
SWE-PolyBench
Fits coding agents whose work mixes bug fixes, features and refactors; borrow the task-category labels and the file and node retrieval metrics as a partial-credit signal.
coding
-
SWE-rebench
Use it when you worry that SWE-bench scores are contaminated: the leaderboard is rebuilt monthly from new issues. Borrow the automated pipeline that turns issues into tested tasks.
coding
-
SWE-smith
Fits teams that need many cheap coding cases from their own repository: point it at a repo, it breaks tests and writes issues. Borrow the bug-injection and keep-only-if-a-test-fails filter.
coding
-
SWR-Bench
Fits pull-request review bots; borrow the structured ground-truth issue list and the coverage-based judge (about 90 percent agreement with humans) as a review rubric.
code review
-
TaxCalcBench
Fits tax-preparation, benefits-eligibility or any form-filling agent whose output is a set of numbered lines. Borrow its line-by-line strict and lenient (plus or minus 5 dollars) scoring, pass^k over repeat runs, and fully synthetic taxpayer cases.
governmentfinancedocuments
-
TeleMath
Fits radio-planning or network-engineering assistants that must do arithmetic on telecom quantities. Borrow its seed-and-expand pattern (50 expert problems grown to 500 with numeric-only answers that grade without a judge).
telecomknowledge qa
-
Terminal-Bench 1.0 (Core v0.1)deprecated
The original terminal task format (task.yaml, Dockerfile, solution.sh, test script) is a clean template for any shell-based agent case; borrow the oracle-solution check.
devopscodingit ops
-
Terminal-Bench 2.0
The standard number for terminal and devops agents; frontier agents were under 65 percent at release. Borrow Harbor's container-per-task layout and the human-written oracle solution.
devopscodingit ops
-
Tool Decathlon (Toolathlon)
Fits agents connected to many SaaS tools over MCP. Borrow the per-task evaluation scripts that check end state in the real app and the practice of seeding realistic initial states.
tool callingworkflow automationemail calendargeneral assistantdevops
-
ToolComp
Fits research-style assistants that chain several tools to reach one answer; borrow its idea of human-corrected step labels (process supervision) next to a final-answer check, and its enterprise versus chat split.
tool callingresearchfinance
-
ToolDial
Fits assistants that must ask for missing information before calling an API. The action vocabulary (request, clarify, fail inform) and the API-graph trick for generating dependent calls are worth copying into a case generator.
general assistanttool callingknowledge qa
-
ToolHop
Fits research or lookup agents that must chain calls where each step depends on the last. Borrow the query-driven method of writing the tools after the question so every case has a verifiable answer.
knowledge qatool callingresearch
-
TutorBench
Fits high-school and AP-level tutoring assistants. Borrow the per-sample rubric pattern, where each prompt ships with its own expert-written checklist that an LLM judge ticks off.
educationknowledge qa
-
UI-CUBE
Fits RPA-style enterprise agents; borrow the split into simple interactions versus multi-step workflows and the resolution and widget-variant sweep.
computer useworkflow automationcrm
-
UltraHorizon
Fits teams stress-testing agents on very long sessions (200k+ tokens, 400+ calls). Borrow the eight-way error taxonomy (in-context locking versus capability gaps).
planninggeneral assistantresearch
-
Vending-Bench
Fits long-running operations agents where the question is coherence over weeks of simulated time. Borrow the single hard metric (ending bank balance) and the failure-mode list (forgotten orders, misread schedules, meltdown loops).
planningfinanceretail
-
WASP
Fits web agents that act on sites with user-generated content. Borrow the two-level metric, intermediate attack success (judge sees the agent diverted) versus end-to-end success (final state check), which separates gullibility from damage.
safetysecurityweb browsing
-
WebArena-Verified
Use instead of original WebArena for any browser agent on internal web apps. Borrow the deterministic, type-aware comparators that replaced substring matching and LLM judging.
web browsingretaildevopsworkflow automation
-
WebChoreArena
Fits back-office browser agents that must aggregate data across many pages. Borrow the memory / calculation / long-term-memory task categories.
web browsingretaildevopsdata analysis
-
WebGames
Fits teams that want fast, hermetic checks of raw browser control. Borrow the password-on-success grading, which needs no judge or state inspection.
web browsinggames
-
WebMall
Fits comparison-shopping agents; borrow the multi-shop offline sandbox built from Common Crawl product pages and the basic / advanced task split.
shoppingretailweb browsing
-
WebWalkerQA
Fits agents that must navigate inside one site (a company portal, a university site, a conference site) rather than search the open web. Borrow the root-site plus question design and the action-count efficiency metric.
web browsingknowledge qa
-
Who&When
Fits teams that want an automated judge to blame the right agent and step after a multi-agent failure. Borrow the agent-plus-step labeling scheme; the reported baselines (about half right on agent, far lower on step) set expectations.
multi agentgeneral assistant
-
WideSearch
Fits agents that build lists or spreadsheets from the web (lead lists, product comparisons, compliance registers). Borrow the table grader: per-column match rules (exact, numeric, date, URL or judge), row and cell F1, and a strict all-cells-correct success rate.
researchweb browsingdata analysis
-
xbench-DeepSearch
Fits Chinese-language search assistants that need a refreshed test set. Borrow the encrypted release with canary strings and the plan to re-issue a new 100-question set periodically.
web browsingknowledge qaresearch
-
τ²-bench
Fits technical-support agents where the customer must act on their own device or account (dual control), and any policy-bound support agent that writes to a backend. Borrow the compositional task generator (atomic subtasks with init, solution and assertion functions), the assertion-based end-state check, and the error split between reasoning and communication.
customer supporttelecomretailtraveltool calling
-
3CB (Catastrophic Cyber Capabilities Benchmark)
Fits offensive-security agents and red-team tooling. Borrow the per-task config that bundles sandbox, prompt elicitations and success rule in one file, and the mapping of tasks to ATT&CK techniques.
securityit ops
-
Agent Security Bench (ASB)
Fits teams comparing defenses across attack stages (query, observation, memory, plan). Borrow the attack-by-stage matrix and the utility-versus-security metric.
safetysecuritytool callingmemory
-
Agent-SafetyBench
Fits general tool-using assistants that need a broad risk taxonomy. Borrow the 8 risk categories and 10 failure modes, and the idea of a small fine-tuned judge (ShieldAgent) for safe/unsafe labels.
safetytool callinggeneral assistant
-
AgentBoard
Fits teams who want partial credit instead of pass/fail for multi-step tasks. Borrow the progress-rate metric (sub-goals reached) and the grounding-accuracy breakdown.
general assistantembodiedgamesweb browsingtool calling
-
AgentClinic
Fits diagnostic or triage assistants that must gather information by asking questions rather than reading a full vignette. Borrow the four-agent setup (patient, doctor, measurement, moderator), the bias injection options, and the structured case JSON.
healthcaremulti agentknowledge qa
-
AgentDojo
Fits assistants that read untrusted content (email, documents, web) and then act with tools. Borrow the pairing of a user task with an injection task, and the utility-under-attack plus targeted attack success rate metrics checked on environment state, not on model text.
safetysecuritytool callingemail calendarfinance
-
AgentGym
Fits teams who want many environments behind one HTTP interface for both training and evaluation of a general agent. Borrow the server-per-environment layout and the AgentEval held-out split.
general assistantembodiedweb browsingtool callinggames
-
AgentHarm
Fits any tool-using assistant that must refuse misuse without losing the ability to do the benign version. Borrow the harmful/benign task pairs and the split scoring (refusal judge, semantic judge, tool-call grading functions).
safetytool callinggeneral assistant
-
Agents4PLC benchmark
Fits PLC or industrial-control code-generation agents. Borrow its three-stage gate (compiles, verifiable, passes formal properties) and its task format of requirement plus human-verified formal spec plus reference Structured Text.
manufacturingcoding
-
Aider polyglot benchmark
Fits chat-style code editing tools that apply diffs to files; borrow the two-attempt protocol (pass rate 1 vs 2) that measures recovery from test feedback.
coding
-
AndroidControl
Fits teams training mobile action models; borrow the paired high-level and low-level instruction per step and the in-domain / out-of-domain split.
mobile
-
AndroidLab
Fits mobile agents that run without network or logins. Borrow the four metrics: success, sub-goal success, reversed redundancy and reasonable operation ratio.
mobile
-
AndroidWorldsaturated
Fits mobile app agents; borrow parameterized tasks whose setup, success check and teardown read the device state, giving unlimited variants of 116 templates.
mobilegeneral assistant
-
API-BLEND
Fits teams fine-tuning a model for API calling in travel or assistant domains. Borrow the three metrics (API-F1, parameter-F1, LCS-F1 for order) and the method of converting existing dialogue corpora into API-call cases.
tool callingtravelgeneral assistant
-
AppWorld
Fits assistants that drive several apps through APIs on a user's behalf. Borrow its database-state unit tests that accept any valid path and also check for unwanted side effects.
general assistanttool callingemail calendarshoppingplanning
-
AssistantBench
Fits research assistants that browse the live web for a user. Borrow the answer-rate versus accuracy split so that abstaining is counted differently from being wrong.
web browsingresearchgeneral assistantshopping
-
AutoDCWorkflow benchmark
Fits data cleaning or data preparation agents that take a messy table and a goal. Borrow its three-way grading: does the clean table answer the purpose, how close are the cell values to the gold table, and how close is the operation sequence to a human workflow.
data analysisworkflow automation
-
B-MoCA
Fits mobile agents that must survive device variation. Borrow the configuration randomizer and the rule-based success detector over system logs, app data and view hierarchy.
mobile
-
BALROG
Fits teams studying long-horizon decision making with partial credit; borrow the per-game progress metrics that give signal even when a run never finishes.
gamesplanning
-
BFCL (Berkeley Function Calling Leaderboard)
Fits any agent whose core job is choosing and filling tool calls; the AST check for single-turn calls and the per-turn state-plus-trajectory check for multi-turn tasks are the patterns to copy, and the leaderboard gives a current baseline for most hosted models.
tool callingmemoryweb browsing
-
BigCodeBenchfrozen
A function-level baseline for library-heavy Python; borrow its high-coverage test writing (about 5.6 tests per task, 99 percent branch coverage).
codingtool calling
-
BLADE
Fits research or analytics agents whose output is an analysis plan, not a single number. Borrow the way expert analysts' decisions (constructs, transforms, model) are collected as ground truth and matched to the agent's choices.
sciencedata analysis
-
BrowserGym / AgentLab
A harness, not a task set; fits teams that want to run several web benchmarks with one agent loop. Borrow the gym-style observation and action space and AgentLab's trace viewer.
web browsingworkflow automation
-
ChemBench
Fits chemistry assistants and lab-support agents that must answer questions and know when they are unsure. Borrow its topic-level reports, numeric-tolerance parsing of answers, and the comparison against a human chemist panel.
scienceknowledge qasafety
-
CliBench
Fits coding, order-set or discharge-summary assistants that output sets of standard codes. Borrow the multi-granular scoring that maps predicted codes to ontology ancestors and reports precision, recall and F1 at each level.
healthcareknowledge qa
-
ClinicalBench (Can LLMs Beat Traditional ML Models in Clinical Prediction?)
Fits teams tempted to use an LLM for risk prediction over structured records. Borrow the matched-cohort comparison against 11 traditional ML baselines and the rule that unparseable answers count as wrong.
healthcaredata analysis
-
Commit0
Fits greenfield coding agents that build from a spec rather than patch; borrow the "percent of tests passed" partial credit and the lint/type-check gates.
coding
-
CORE-Benchsaturated
Fits reproducibility or analysis agents; borrow the question-and-answer report.json grading that avoids judging code directly.
researchcodingdata analysis
-
CorpFin (Vals AI)saturated
Fits long-document review agents in corporate lending or legal operations. Borrow the question categories (extraction, interpretation, cross-reference, numeric reasoning, market standard) when writing cases over 200-page contracts.
financelegaldocumentsknowledge qa
-
CRAB
Fits agents that must coordinate across a phone and a computer. Borrow the graph evaluator that gives partial credit for sub-goals reached in any valid order.
computer usemobilemulti agent
-
CRAG (Comprehensive RAG Benchmark)
Fits retrieval-augmented question answering agents that call search or knowledge APIs. Borrow its mock-API sandbox (fixed search results so runs are repeatable) and its scoring that gives +1 for correct, 0 for missing and -1 for hallucinated answers.
knowledge qaresearchtool calling
-
CRMArena
Fits agents that read a real CRM (Salesforce or similar) to answer service, analyst or manager questions. Borrow the persona-based task taxonomy, the 30% unanswerable queries, and the way a synthetic org is generated from a schema with hidden latent variables that drive the ground truth.
crmcustomer supportknowledge qatool calling
-
CRUXEvalfrozen
Fits teams checking whether a model actually reads code before editing it; borrow the execution-prediction task as a cheap probe inside a bigger code-review suite.
coding
-
Cybench
Fits offensive-security or CTF-style agents and teams that want a known capability baseline. Borrow the subtask mode that gives partial credit for intermediate steps and the per-task human solve time as a difficulty scale.
securitycoding
-
DA-Codefrozen
Fits data-analysis or notebook agents that write and run code over files. Borrow its three task categories (data wrangling, machine learning, exploratory analysis), the Docker sandbox and the evaluate.py comparison of output files to gold files.
data analysiscodingml engineering
-
DevBench (DevEval)frozen
Fits agents covering more than coding, such as design docs and test writing; borrow its lifecycle stages as a taxonomy for an agent's golden set.
coding
-
DiscoveryBench
Fits research agents that must turn a dataset and a goal into a written finding. Borrow the faceted judge: split gold and predicted hypotheses into context, variables and relationship and score each part separately.
sciencedata analysisresearch
-
DocBench
Fits document question-answering assistants over contracts, filings or papers. Borrow the four question types (including unanswerable) over 229 real documents in five domains.
documentsknowledge qafinancelegal
-
DSBench
Fits data science agents that must read long briefs and large or multi-table files, and agents that build models end to end. Borrow the relative performance gap metric that scores a model against the human leaderboard on the same Kaggle task.
data analysisml engineeringcoding
-
DSEval
Fits notebook-style assistants where each question builds on the last. Borrow the problem set format (ordered problems sharing one session) and the bootstrapped annotation method for expanding cases.
data analysiscoding
-
EHRAgent (EHRSQL-derived agent evaluation)
Fits data-retrieval agents that let clinicians ask questions of structured EHR tables in plain language. Borrow the code-as-plan action format, the interactive debugging loop, and the question complexity levels (number of tables joined).
healthcaresqldata analysistool calling
-
EHRNoteQA
Fits chart-summarization or note question-answering assistants that read long discharge summaries. Borrow the clinician-reviewed question generation over multiple notes and the reported correlation with clinician judgments as a way to validate a proxy benchmark.
healthcaredocumentsknowledge qa
-
FinBen
Fits teams that need a broad baseline for a finance language model behind an agent rather than the agent itself. Borrow its grouping of 36 datasets into 24 task types to check which skills your model is missing.
financeknowledge qadata analysis
-
FRAMES
Fits research or multi-hop retrieval agents that must chain facts across documents. Borrow its per-question reasoning-type labels and the inspect_evals Socrates scoring, which discounts accuracy by extra retrieval hops and penalizes requests for documents that do not exist.
knowledge qaresearch
-
FunctionChat-Bench
Fits chat assistants where every turn must be one of call, answer, ask, or refuse. The four-way output taxonomy and the rubric-based pass/fail judge are a compact grading design to reuse.
general assistanttool calling
-
GDM Dangerous Capabilities: Capture the Flag
Fits offensive-security agents and teams that need a small in-house CTF set that is not on the public internet. Borrow the attacker-plus-victim compose layout and the at-least-one-of-N epochs reducer.
securityit ops
-
GDM Dangerous Capabilities: Self-proliferation
Fits teams giving an agent real web, cloud and shell access who need to know how far it can get on its own. Borrow the milestone mode that scores partial progress and the human-approval gate on every tool call.
safetysecurityweb browsingit ops
-
GTA (General Tool Agents)
Fits assistants whose users send screenshots, photos or files and expect the right tools to be picked without being told. Borrow the human-written implicit-tool-use queries and the split between step-level tool accuracy and final-answer accuracy.
tool callinggeneral assistantdata analysis
-
HammerBench
Fits on-device or app assistants doing slot filling. Borrow the per-turn snapshot metrics (parameter hallucination rate, parameter missing rate, progress rate) and the four perturbation types (imperfect, Q&A trajectory, intent shift, external reference).
mobilegeneral assistanttool calling
-
HR-MultiWOZfrozen
Fits HR help-desk chatbots that need conversation test cases. Borrow the LLM-generated dialogue pipeline built to avoid real HR conversations.
hrcustomer support
-
IaC-Eval
Fits infrastructure-as-code agents; borrow intent specs checked with OPA against the Terraform plan instead of string-matching the HCL.
devopsit ops
-
InfiAgent-DABench
Fits a data analysis assistant that answers questions over uploaded CSV files with code. Borrow the trick of rewriting open-ended questions into closed-form answers so grading is an exact match.
data analysiscoding
-
InjecAgentfrozen
Fits a first, cheap check of any tool-calling agent for indirect prompt injection. Borrow the two attacker goals (direct harm, data theft) and the user-case times attacker-case crossing that gives many cases from few templates.
safetysecuritytool calling
-
LAB-Bench
Fits research-assistant agents for biology that answer questions over papers, databases and sequences. Borrow the eight-category taxonomy (LitQA2, DbQA, SuppQA, FigQA, TableQA, ProtocolQA, SeqQA, Cloning) and the accuracy-plus-coverage reporting that rewards abstaining.
scienceknowledge qaresearch
-
LegalAgentBench
Fits legal research or due-diligence agents that query several databases and write up the result. Borrow its progress-rate grading (keyword checks on annotated intermediate steps) and its tool-over-corpus sandbox.
legaltool callingknowledge qadocuments
-
LiveCodeBench
A model-level coding baseline rather than an agent test; borrow the date-windowed release scheme that lets you report only problems newer than a model's cutoff.
coding
-
LoCoMo
Fits assistants that must remember facts a user told them weeks ago across sessions. Borrow the persona-plus-event-graph recipe for generating long, consistent fake conversations without real user data.
memoryknowledge qa
-
Long Code Arenafrozen
Six project-wide tasks in one suite; the CI build repair task, graded by re-running the real GitHub Actions workflow, is a usable pattern for devops agents.
codingdevops
-
LongBench v2
Fits any agent that must read long inputs (8k to 2M words) before answering, such as contract review or codebase question answering. Borrow its six-category task split and its human-written, time-limited question style, which keeps questions hard to shortcut.
documentsknowledge qacodingresearch
-
LongMemEval
Fits chat assistants with long-term memory across sessions; the five ability categories (extraction, multi-session reasoning, temporal, updates, abstention) are a ready taxonomy for a memory test set. Borrow the needle-in-history construction and the 30 abstention cases.
memoryknowledge qageneral assistant
-
m&m'sfrozen
Fits agents that plan a short pipeline over heterogeneous tools (OCR, captioning, weather, search) before executing. Borrow its plan-level metrics (tool-F1, argname-F1, pass rate) and its finding that step-by-step planning with feedback beats one-shot plans.
tool callingplanningdocumentsgeneral assistant
-
Make Me Pay
Fits payment or negotiation agents that talk to other agents or users and must not be manipulated (or manipulate). Borrow the two-model setup where the outcome (money moved) is checked programmatically rather than judged.
multi agentsafety
-
MedCalc-Bench
Fits clinical decision-support assistants that must compute scores or doses from notes. Borrow the tolerance-band grading (plus or minus 5 percent for decimal outputs), the relevant-entities field for partial-credit diagnosis, and the 55 calculator taxonomy.
healthcareknowledge qadata analysis
-
MedChain
Fits end-to-end clinical workflow agents where an early mistake (wrong specialty, missed question) should hurt later stages. Borrow the five-stage sequential case format and the per-stage metrics (accuracy and IoU for referral and history, LLM scoring for diagnosis).
healthcaremulti agentknowledge qa
-
Mind2Web-SC
Fits teams adding a policy guardrail in front of a web or booking agent. Borrow the rule-plus-profile case format and the split accuracy (right decision, right violation named, right action kept).
safetyweb browsing
-
MLE-bench
The standard test for ML-engineering agents; borrow medal-threshold grading against a human leaderboard and the 22-competition Lite split for cheaper runs.
ml engineeringdata analysis
-
MTU-Bench
Fits booking and lookup agents built on slot-filling dialogues. Borrow its 2x2 taxonomy (single or multi turn by single or multi tool) and its metric set (tool selection, parameter accuracy, turn success, task progress) that needs no LLM judge.
travelgeneral assistanttool calling
-
MultiHop-RAG
Fits retrieval-augmented agents over a corpus where one question spans several documents. Borrow its split of retrieval scoring (did the right evidence come back) from answer scoring, and its four query types including null queries with no answer in the corpus.
knowledge qaresearch
-
NESTFUL
Fits agents that must pass results between tools in one plan (data pipelines, calculators, code helpers); borrow its label-and-reference format for nested calls and its metric ladder (API-name F1, parameter F1, partial and full sequence match, executed win rate).
tool calling
-
NoisyToolBench
Fits any agent that gets vague requests and should ask rather than guess. Borrow the case format (imperfect query, expected clarifying question, scripted answer, expected call) and the ToolEvaluator that scores both accuracy and redundant questions.
tool callinggeneral assistant
-
OfficeBench
Fits office assistants that move content between documents, spreadsheets, email and calendars. Borrow the three-tier design (one, two, three apps) and the mix of exact, fuzzy and execution-based checks on output files.
documentsspreadsheetsemail calendargeneral assistant
-
OmniACTfrozen
Fits teams training grounding-plus-planning in one shot. Borrow the sequence score and action score (click-in-box, key set, typed-text BLEU).
computer useweb browsing
-
OmniDocBench
Fits the parsing step of any document pipeline, where PDFs become text for an agent to reason over. Borrow its per-element metrics (edit distance for text, TEDS for tables, CDM for formulas) and its page attribute labels for slicing results by layout and language.
documents
-
OSWorlddeprecated
The reference for desktop agents; borrow the per-task setup config plus execution-based evaluator script pattern and the VM snapshot reset.
computer usedocumentsspreadsheetsit ops
-
QRData
Fits analysis assistants that must apply statistics or causal methods correctly, not just run code. Borrow the pairing of each question with a data sheet and a companion text-only set to tell reasoning failures from data-handling failures.
data analysisscienceeducation
-
RAGBench
Fits teams building the judge for a retrieval-augmented agent rather than the agent itself. Borrow its TRACe labels (relevance, utilization, adherence, completeness) at sentence level as a rubric for your own answer grader, and use its 12 sub-datasets to test that grader.
knowledge qafinancehealthcarelegalcustomer support
-
RE-Benchdeprecated
Fits ML research-engineering agents; its value is the 71 human-expert 8-hour attempts that give a time-budgeted human baseline to compare against.
ml engineeringresearch
-
RoTBench
Fits teams whose tool schemas change or are poorly documented. The pattern to copy is taking a clean case set and producing slight, medium, heavy and union noise variants to measure robustness.
tool callinggeneral assistant
-
SciCode
Fits scientific-coding assistants; borrow the main-problem versus subproblem scoring so partial progress is visible.
sciencecoding
-
ScienceAgentBench
Fits data-analysis or scientific coding agents that take a dataset and a goal and must hand back a runnable program. Borrow its per-task evaluation scripts that execute the generated code and compare outputs, plus the best-of-three-runs scoring.
sciencedata analysiscodingresearch
-
SciKnowEval
Fits science question-answering assistants in biology, chemistry, physics or materials. Borrow the five-level taxonomy (memory, comprehension, reasoning, discernment, application) as a way to bucket an agent's own test cases by depth.
scienceknowledge qasafety
-
Seal-Toolsfrozen
Fits teams who need a large, fully made-up tool catalogue for fine-tuning or retrieval tests; borrow the self-instruct pipeline (field, tool, instance) and its format accuracy plus tool and parameter precision/recall/F1 scoring.
tool calling
-
ShortcutsBenchfrozen
Fits workflow-automation agents that compose many app or SaaS APIs into one long sequence. Borrow its four-way grading (API choice, parameter fill, reuse of prior outputs, asking for input) and its difficulty tiers by sequence length.
workflow automationtool callingmobileplanning
-
SimpleQA
Fits assistants whose risk is confidently stating wrong facts. Borrow its three-way grade (correct, incorrect, not attempted) and its F-score that rewards abstaining over guessing; this is a good pattern for any agent that should say "I don't know".
knowledge qa
-
Spider 2.0
Fits analytics or warehouse agents that must answer questions over big real schemas in cloud SQL dialects. Borrow its three sub-variants: Spider2-Lite (547 tasks, BigQuery 214, Snowflake 198, SQLite 135), Spider2-Snow (547 tasks on Snowflake only) and Spider2-DBT (68 repository-level dbt tasks on DuckDB), and its execution-result grading.
sqldata analysiscoding
-
Spider2-V
Fits agents that operate data tools through a GUI or mixed GUI plus code, such as a pipeline operator or warehouse admin assistant. Borrow its per-task evaluation functions and the VM built on OSWorld with human-validation scripts.
computer usedata analysisdevopssql
-
SpreadsheetBench
Fits spreadsheet or Excel assistants that edit workbooks from a user request. Borrow the online-judge pattern: about three input workbooks per instruction, and the output cells must match on every one.
spreadsheetsdata analysiscoding
-
ST-WebAgentBench
Fits browser agents that work inside company systems under rules, such as CRM or admin-panel automation. Borrow the Completion Under Policy metric (a task only counts if no policy was broken), the six policy dimensions, and the per-policy evaluator pattern.
safetyweb browsingcrmdevops
-
StableToolBench
Fits agents that call many third-party web APIs and need a replayable test bed; the cache plus LLM-simulated-API server is the pattern to copy when the real APIs are flaky, and the solvable-only filtering shows how to prune unanswerable cases.
tool callinggeneral assistant
-
SWE-bench Litefrozen
A cheap smoke test for a coding agent: single-file edits, at most three hunks, no images or links in the issue. Borrow its filter rules when you need a small, clean subset of a bigger set.
coding
-
SWE-bench Multimodal
Fits agents that work on front-end or visualization code where the bug report is a picture. Borrow the way image inputs are attached to an otherwise text-only task.
coding
-
SWE-bench Verified
The default number to compare against for issue-to-patch coding agents in Python. Borrow its human screening step, where engineers rejected tasks whose issue text or tests were unfair.
coding
-
SWE-bench-javafrozen
An early Java-only issue set; useful if your agent targets JVM codebases and you want a tiny hand-verified pilot. Borrow the Docker-per-instance build setup for Maven projects.
coding
-
SWE-Gym
A training environment rather than a test set; fits teams fine-tuning their own coding agent or verifier. Borrow the prebuilt per-instance Docker images and the Lite split of 234 tasks.
coding
-
TableBench
Fits assistants that answer questions over a single table in a chat. Borrow the 18-category taxonomy under four headings and the per-category metric choice (exact match, exact match with a 10 percent margin, ROUGE-L, pass at 1 for charts).
data analysisknowledge qa
-
TheAgentCompany
Fits digital-worker agents that touch several internal tools and must ask coworkers for details. Borrow the self-hosted company sandbox, the LLM-played colleagues and the checkpoint-plus-result scoring.
general assistantcodinghrdocumentsworkflow automation
-
ToolEyes
Fits general assistants with a broad tool library. Borrow the five-dimension scoring (format, intent, planning, tool selection, answer) that mixes rule checks with a judge instead of grading only the final answer.
general assistanttool callingfinanceresearch
-
ToolSandbox
Fits assistant agents whose tools have hidden state dependencies (one call only works after another). Borrow the milestone and minefield grading: required world-state checkpoints in a partial order, forbidden events that zero the score, and similarity scoring of each checkpoint.
tool callinggeneral assistantmobile
-
TravelPlanner
Fits planning assistants that must juggle many constraints. Borrow the split between commonsense and hard constraint pass rates plus a final all-constraints pass rate.
travelplanningtool calling
-
UltraToolfrozen
Fits agents that plan multi-step work over a large tool catalog, and teams that want to grade planning separately from execution. Borrow its planning/creation/usage split, its six-dimension GPT-4 plan rubric, and its key-value accuracy for tool selection and arguments.
tool callingplanninggeneral assistant
-
USACO
A cost-tracked algorithmic baseline available on HAL; useful as a sanity check for reasoning-heavy coding agents, not for product engineering.
coding
-
VisualWebArena
Fits agents that must read screenshots or product images while browsing (classifieds, marketplaces). Borrow the image-conditioned task templates and the execution-based checks.
web browsingretailshopping
-
WebCanvas (Mind2Web-Live)
Fits any live-site browser agent where the final page cannot be checked directly. Borrow key-node grading: credit for reaching required URLs or elements mid-task.
web browsingshoppingtravel
-
WebLINX
Fits chat-driven browser copilots where a human gives instructions turn by turn. Borrow the demonstration format of paired user utterances and browser actions.
web browsinggeneral assistant
-
WebVoyager
Fits consumer-facing browser agents on well-known sites. Borrow the screenshot-based auto-judge that reads the last k screenshots.
web browsingshoppingtravelknowledge qa
-
Windows Agent Arena
Fits agents that must run on Windows desktops. Borrow the Azure-parallel runner that finishes the benchmark in about 20 minutes.
computer usedocumentsit ops
-
WorfBench
Fits agents that produce a plan before executing, such as workflow builders. Borrow the graph matching evaluation (F1 on node chain and on the workflow graph) that rewards correct parallel structure, not just step order.
planningtool callingembodiedworkflow automation
-
WorkArenasaturated
Fits browser agents working inside ServiceNow or similar ITSM/HR portals. Borrow the task templates (33 atomic tasks instantiated into thousands of variants) and the per-task validate() reward on a live instance.
it opsworkflow automationweb browsingcomputer use
-
WorkArena++
Fits browser agents doing compositional work in ITSM and HR portals. Borrow its way of building hard tasks by composing atomic ones and its ground-truth trace generator for training data.
it opsworkflow automationweb browsingplanning
-
WorkBench
Fits assistants that act on email, calendar and CRM data through tools. Borrow the outcome-centric grading that compares final database state regardless of action order and reports side effects separately.
email calendarcrmdata analysistool callingworkflow automation
-
τ-benchdeprecated
Fits support or booking agents that must follow a written policy while talking to a customer and writing to a backend. Borrow the end-state database diff as the pass/fail check, the pass^k reliability metric, and the simulated user prompted with a hidden user instruction.
customer supportretailtraveltool calling
-
AgentBench
Fits teams who want one harness covering shell, database and web-style tasks to sanity-check a general agent. Borrow the containerized per-environment servers; the 2025 function-calling version keeps it usable with current tool-call APIs.
general assistantit opssqlshoppinggames
-
AgentVersefrozen
Fits teams prototyping multi-agent orchestration rather than teams looking for a test set; the repo is a framework with a HumanEval runner, useful as a reference for recruit-plan-act-review loops.
multi agentcodinggeneral assistant
-
API-Bankfrozen
Fits assistants that must pick one API out of many and fill its arguments from a chat history. Borrow its three-level split (call, retrieve then call, plan then retrieve then call) and its dialogue-plus-expected-call case format.
tool callinggeneral assistant
-
BIRD
Fits single-shot text-to-SQL agents that answer questions over one database with a known schema. Borrow its execution-accuracy grading, the evidence hint pattern and the human baseline (92.96 execution accuracy) as a reference point.
sqldata analysisknowledge qa
-
ClassEvalfrozen
Fits code generators that emit whole modules; borrow class-level versus method-level pass@1 reporting to show where multi-part outputs break.
coding
-
DocILEfrozen
Fits invoice and purchase-order extraction agents. Borrow its 55-class field taxonomy, its line-item grouping task, and its scorer that matches predicted fields to ground truth by bounding-box overlap and reports precision, recall and F1 per field.
documentsfinanceworkflow automation
-
DocMath-Eval
Fits finance or analyst agents that must compute figures from 10-K and 10-Q filings. Borrow its four-way split (simple vs complex reasoning, short vs long document) and its numeric answer checker that accepts chain-of-thought or program-of-thought output.
financedocumentsdata analysis
-
FinanceBench
Fits retrieval-augmented finance assistants that answer questions over company filings. Borrow its case format (question, answer, evidence string, justification, question type, reasoning type) as a template for a golden set over your own document corpus.
financedocumentsknowledge qa
-
GAIAsaturated
Fits research-style assistants that must return one checkable answer. Borrow the short-answer exact-match format and the three difficulty levels.
general assistantresearchweb browsingknowledge qa
-
Gorilla APIBenchfrozen
Fits code-writing agents that must call library APIs from documentation; borrow its AST sub-tree match (compare only the arguments that matter) and its separate count of hallucinated versus wrongly used APIs.
tool callingml engineeringcoding
-
HumanEvalFix (HumanEvalPack)frozen
A tiny bug-fix template: 164 HumanEval problems times six languages with planted bugs; useful only as a quick multi-language smoke test for a repair agent.
coding
-
InterCode CTFsaturated
Fits teams that want a quick, cheap smoke test for a shell-using agent. Borrow the environment-as-RL-loop design where code is the action and execution output is the observation.
securitycoding
-
LawBenchfrozen
Fits Chinese-law question-answering or document-processing assistants. Borrow its three-level split (memorize, understand, apply) and its abstention-rate metric for answers that cannot be parsed.
legalknowledge qadocuments
-
LegalBench
Fits contract-review, compliance and legal-research assistants that must classify clauses or apply a rule to facts. Borrow its six-way reasoning taxonomy (issue, rule, conclusion, application, interpretation, rhetoric) and its short hand-written task cards.
legalknowledge qadocuments
-
MetaToolfrozen
Fits any agent with a large tool catalog where the first failure is picking the wrong tool. Its four selection subtasks (similar, scenario-specific, reliability, multi-tool) are a ready taxonomy for tool-routing test cases.
general assistanttool calling
-
Mind2Web
Fits teams training or testing step-level action prediction on real site layouts. Borrow the cross-task / cross-website / cross-domain split design.
web browsingshoppingtravelgeneral assistant
-
MINTfrozen
Fits agents that iterate with a user on code or quantitative answers. The useful part is the GPT-4 feedback simulator and the turn-budget analysis (k = 1 to 5 turns), which shows how much a feedback loop is worth.
codinggeneral assistanttool callingembodied
-
RepoBenchfrozen
Fits autocomplete or inline-suggestion products, not agents; borrow exact match plus edit similarity plus CodeBLEU as cheap text-level grading for completion cases.
coding
-
RestBenchfrozen
Fits agents that call real REST services from OpenAPI docs. Borrow the small hand-annotated gold-path format and the correct-path-rate metric alongside success.
tool callinggeneral assistantknowledge qa
-
SheetCopilotfrozen
Fits agents that drive a spreadsheet application through a fixed action set rather than writing code. Borrow the per-task checking file that names which sheet properties to compare against one or more reference workbooks.
spreadsheetsworkflow automation
-
SWE-benchfrozen
Fits any agent that fixes bugs or implements changes in an existing codebase from a written issue. Borrow the FAIL_TO_PASS / PASS_TO_PASS test split and the per-task Docker image pattern.
coding
-
Sycophancy Eval
Fits any agent that answers users who push back, such as support, tutoring or advisory agents. Borrow the "are you sure?" follow-up turn and the three metrics (confidence, apologize rate, truthfulness) as a cheap regression check for caving to pressure.
safetyknowledge qageneral assistant
-
T-Evalfrozen
Fits teams that want to diagnose which step of tool use their agent gets wrong (format, tool choice, plan, reasoning, argument extraction, response review) rather than only the end result. Borrow its decomposition and its per-step graders.
tool callingplanning
-
TaskBenchfrozen
Fits workflow-automation agents that must plan a tool chain before executing. Borrow the node-F1 and edge-F1 scoring of predicted tool graphs and the back-instruct data generator.
tool callingplanningworkflow automation
-
TeleQnA
Fits network-operations or telecom customer-support assistants as a knowledge check before agent testing. Borrow its five-category split (lexicon, research overview, research publications, standards overview, standards specifications).
telecomknowledge qa
-
ToolAlpacafrozen
Fits teams that need to test an agent against APIs that do not exist yet or cannot be hit in test; borrow its LLM-simulated API server and its procedure/response/overall grading split. Mostly a training corpus, not a benchmark.
tool callinggeneral assistant
-
ToolBench (ToolLLM)frozen
Fits agents that must choose from a very large tool catalogue and chain real web APIs; borrow its single-tool / same-category / cross-collection split (G1, G2, G3) for building a tool coverage taxonomy.
tool callinggeneral assistant
-
ToolEmufrozen
Fits teams that need to test dangerous actions before any real tool exists. Borrow the LLM-emulated tool sandbox and the paired safety and helpfulness scores on a 0-3 scale.
safetytool callinggeneral assistantfinance
-
ToolQAfrozen
Fits question-answering agents over a company's own tables or document store. Borrow its template-plus-sampled-values way of generating questions with programmatically computed answers, and its check that the question cannot be answered without the tool.
knowledge qatool callingdata analysissql
-
ToolTalkfrozen
Fits personal-assistant agents that act on calendars, email and messages in a conversation. Borrow the simulated tool implementations with mock databases and the strict success rule (perfect recall of required calls, zero wrong actions).
general assistantemail calendartool calling
-
WebArena
Fits browser agents that act inside internal web apps (shop admin, issue tracker, forum). Borrow the self-hosted site sandbox and the per-task evaluator mix of exact answer, target URL and page-content program checks.
web browsingretaildevopsworkflow automation
-
DS-1000
Fits a code assistant that writes pandas or numpy snippets for analysts. Borrow the two-part check (run on fresh inputs against a reference solution, plus a token check on the code text) and the practice of perturbing scraped problems to limit memorization.
data analysiscoding
-
EHRSQL
Fits text-to-SQL or data-retrieval agents over hospital databases. Borrow the questions collected from 222 real hospital staff, the time-expression templates, the unanswerable question split, and the execution-based F1 that penalizes confident wrong answers.
healthcaresql
-
ScienceWorld
Fits teams who want a procedure-following environment with partial-progress scoring; the 30 task types have about 7,000 parametric variations, so it resists memorization better than ALFWorld.
scienceeducationembodiedplanning
-
WDC Products
Fits product catalog or deduplication agents that must say whether two records are the same item. Borrow its difficulty dials (share of corner cases, share of unseen entities, training set size) across 27 variants built from 11,715 offers of 2,162 products.
data analysisretailshopping
-
WebShopfrozen
Fits shopping assistants; borrow the self-hosted product catalog sandbox and the attribute-and-price reward computed from the bought item.
shoppingretailweb browsing
-
APPSfrozen
Still used as a large, cheap code-generation baseline; its test-case checker is reused by LiveCodeBench, so it is a known-good grading script.
coding
-
FinQA
Fits finance agents that compute figures from reports and must show their work. Borrow its two-level grading (execution accuracy on the final number plus program accuracy on the reasoning steps) when you want to check reasoning, not just the answer.
financedocumentsdata analysis
-
TAT-QA
Fits agents that read financial statements where the answer needs both a table and the surrounding text. Borrow its answer-type tagging (span, multi-span, count, arithmetic) and its exact-match and F1 scorer with scale handling.
financedocumentsdata analysis
-
ALFWorldsaturated
Fits teams who need a cheap, deterministic multi-step environment for smoke tests of a planning loop; it is also the base of the household tasks in AgentBench, AgentBoard and AgentGym.
embodiedplanning
-
BugsInPyfrozen
A Python counterpart to Defects4J for repair agents; useful as a small, reproducible pilot set before building your own.
coding
-
DocVQA
Fits agents that read scanned or photographed documents (forms, letters, reports) and pull out a short answer. Borrow its question-over-image case format and its ANLS scoring, which tolerates small OCR spelling errors instead of demanding an exact string.
documents
-
MedQA (USMLE)saturated
Context only. Fits nothing agentic; useful as a knowledge floor for a medical assistant and as the source vignettes that AgentClinic and MedCalc-Bench reuse.
healthcareknowledge qa
-
CORDfrozen
Fits receipt and expense capture agents. Borrow its two-level label scheme (five superclasses, 42 subclasses) and its line-group annotations as a template for structured extraction targets.
documentsretail
-
Magellan / DeepMatcher entity matching datasetsfrozen
Fits record linkage or deduplication agents that need labeled pairs to test on across domains. Borrow its three flavors (structured, textual, dirty) and its 3:1:1 splits as a template for building a matching test set.
data analysis
-
Defects4Jfrozen
Still the reference set for Java program-repair agents (RepairAgent, ReinFix report on it); borrow the minimized single-commit fix and triggering-test rules for writing repair cases.
coding
No benchmark matches.
From references/benchmarks.yaml in aot-evals 1.0.1. Missing a benchmark? Tell us.