# AOT Evals: the method (Mahal Systems)

AOT Evals is an open-source method, and a Claude Code plugin that runs it, for designing, building, calibrating and running evals for a production agent, so the tests predict how it does in production and show what to fix next.

Cases come from the agent's real traffic; outcomes are graded with code first and calibrated judges second; correctness, operational cost and trust in the instruments are reported separately; and the report refuses any number it can't defend, naming the gate (G1-G14).

Version 1.0.1. Everything in one Markdown file: https://mahalsystems.ai/aot/evals/llms-full.txt · As JSON: https://mahalsystems.ai/aot/evals/index.json · Download: https://github.com/Mahalanobis-Systems/aot-evals/releases/download/aot-evals--v1.0.1/aot-evals-method-1.0.1.zip · Plugin: https://github.com/Mahalanobis-Systems/aot-evals

## Pages

- [The full reference](https://mahalsystems.ai/aot/evals/method/): principles, measurement frame, repository contract, commands, judges, measurements, experiments, report.json, gates
- [The process](https://mahalsystems.ai/aot/evals/process/)
- [Choosing an eval strategy from an agent's properties](https://mahalsystems.ai/aot/evals/process/agent-properties-framework/)
- [Steps 1–4: Understand the system](https://mahalsystems.ai/aot/evals/process/phase-1-understand-the-system/)
- [Steps 5–7: Understand the usage](https://mahalsystems.ai/aot/evals/process/phase-2-understand-the-usage/)
- [Steps 8–12: Build the evals](https://mahalsystems.ai/aot/evals/process/phase-3-build-the-evals/)
- [Principles](https://mahalsystems.ai/aot/evals/process/principles/)
- [Practice findings](https://mahalsystems.ai/aot/evals/practices/): what the 2025-2026 literature says about evaluating production agents
- [Benchmark catalog](https://mahalsystems.ai/aot/evals/benchmarks/): 364 public agent benchmarks

## Commands (each with its exit gate)

- C0 import: Import your data as an export (trace dump, dataset download, CSV/JSONL from your observability stack) into data/, with an export.manifest.json entry: source, query, window, row count, date. Exit gate: Every downstream number traces to an export manifest entry; no command calls a live vendor API.
- C1 inventory: Write agent.yaml: surface, triggers, instructions, tools (credential names only), harness, the model actually used per call site, version history, the stable window, and the invocation command. Exit gate: Stable window stated with evidence; model per call site observable; invoke.cwd exists; the agent runs once end to end from its command.
- C2 taxonomy: Build the coverage list from a source outside the suite: the product spec, API/tool surface, help-centre topics or job description. The owner edits it; the source document is recorded. Exit gate: taxonomy.yaml names a source outside cases/ and a date, and its commit precedes the first case.
- C3 categorize: Sort traces into categories, one per trace. Give each a worth and a per-failure-kind cost matrix. Count observed share. Exit gate: Every category has a definition, an example, worth, a simple flag, a cost row and a measured share. "Other" is under 10%.
- C4 decompose: Within each category: feature → scenario → failure mode. Mark each failure mode code-checkable, judge-needed or monitor-only, and set plane-B budgets. Exit gate: Every failure mode names its category, check type and cost cell; budgets.yaml covers every category.
- C5 allocate: Size both case sets and print the arithmetic. Error-finding: ≥10 per category, 3 for simple ones. Quality-estimate: proportional to observed share, sized by the interval the owner needs (M14). Exit gate: Both budgets printed with target intervals and the minimum detectable effect (M7); every cell filled, scheduled, or declared out of scope with a reason.
- C6 author: Write the labelling standard, then cases: frozen inputs, start state, trigger, scripted follow-ups, expected end state, and requirements walked rule by rule from the standard. Prefer self-verifying pairs where state allows. Scrub before writing. Exit gate: The standard exists; every case has a set, a category, failure modes, provenance, requirements covering every rule that applies to it, and no secrets.
- C7 screen: Per case: a human read; the checks must pass on start → expected end (reference) and fail on start → start (do nothing); a third pass, do-nothing plus a success claim, catches checks that trust the agent's report (G10). With a reference answer: other correct wordings must pass, the reference answer with nothing changed must fail, and an answer with every number wrong is reported if it passes. Drop cases that fail, and record the drop. Exit gate: Screens recorded per case; drop rate reported.
- C8 checks: Implement plane-A checks: code state diffs first, graded against the goal rather than one path; binary judges only where code cannot decide; side-effect checks on every agent; a grounding check for every fact a prose answer states. A check never uses the agent's own report of success. Exit gate: Every failure mode has at least one binary check; side-effect checks present; every requirement names a check that runs on its case, or why it is unchecked.
- C9 label: Collect human labels per judge, stratified across categories: one item per case first, then other trials where the answers differ. Target 100–200; below 60 the interval is too wide to act on. Exit gate: Label count and stratification printed per judge.
- C10 calibrate: Measure each judge against its labels: TPR and TNR separately (never raw agreement), κ, confusion matrix, cost per judgement. Choose a probabilistic backend's threshold on the labels. Version the criteria. Exit gate: Every judge has TPR/TNR on ≥60 labels at or above its floors, a named backend and model family, and a version. Uncalibrated judges may run but never count toward headline numbers.
- C11 reliability: Run m fresh sessions per item on two backends from different model families; report agreement at three levels (all / majority / none) next to C10's TPR/TNR. Exit gate: Self-consistency and cross-family agreement reported next to TPR/TNR, never in place of it.
- C12 baseline: Run the suite k = 3–5 times on an unchanged, committed build. Write per-trial outcomes and plane-B counters. Compute the variance band. Exit gate: Run complete and finished, manifest complete, no eval-file exposure, variance band per category, plane B recorded against budgets.yaml.
- C19 agreement: Runs after C12 (numbered last so earlier numbers stay stable). A person labels whole baseline runs pass or fail with a note, seeing the request, the answer and what changed, never the checks. Every labelled run is re-graded from its raw record with the suite as it is now, so labels outlive suite versions. Report the catch rate and clean pass rate (M29), dev and holdout; give every dev disagreement a cause. Exit gate: ≥60 labelled runs with ≥15 of each kind; a measurement current for the suite and the labels; dev catch and clean pass rates at or above their floors (85% by default); every dev disagreement triaged. Until then G14 fails and comparisons are provisional.
- C13 compare: Run a candidate on the same cases with one declared independent variable. Report gains, losses and a paired test, never a net delta alone, with plane-B deltas alongside. Exit gate: Paired test present; gains and losses listed; manifests differ only on the declared variable.
- C14 coverage: Count cases per (taxonomy entry × failure mode), zeros included, with the source document named. Exit gate: Coverage reported against an external list, or refused (G5).
- C15 redundancy: Over ≥2 agent versions, classify each case; report the discriminating fraction and a per-case index; propose prunes and cells to extend. Exit gate: Discriminating fraction published; below the threshold (default 0.3), a prune/extend proposal.
- C16 drift (guidance only): Re-run the frozen set on a schedule and on every prompt, skill, tool, model or harness change, against the baseline's band. Exit gate: Guidance only; not yet automated.
- C17 online (guidance only): Set up the online regime: shadow → canary → online judging; behavioural signals; fingerprints. Exit gate: Guidance only; not yet automated.
- C18 report: Write report.json (§8), then the HTML report (§9). Exit gate: Every gate's status is in the report.

## Gates (what the report refuses, and when)

- G1: when No condition record (model ids, commits, harness, k), refuses the whole report
- G2: when k = 1, refuses any claim of improvement, regression or drift
- G3: when Judge uncalibrated (under 60 labels, below its floors), or reported as raw agreement, refuses that judge's contribution to any headline; the judge is greyed
- G4: when Generative judge from the same model family as the agent, for pairwise or preference grading, refuses the pairwise result. A System One judge satisfies this
- G5: when Coverage list with no external source, derived from the suite's own cases, or committed after them, refuses the word "coverage"; prints MISSING and the reason
- G6: when Headline pass rate from the error-finding set, refuses the word "quality"; offers the weighted estimate instead
- G7: when Comparison without a paired test, or with undeclared manifest differences, refuses the comparison
- G8: when Cases that have not passed the reference / do-nothing screen, refuses their inclusion in any gated number
- G9: when Simulated user present, results not reported per simulator, refuses the aggregate score
- G10: when A check that relies on the agent's own report of success, refuses the check
- G11: when A "better" verdict while plane B regressed beyond budget, refuses the verdict; it becomes "trade", with both numbers
- G12: when A number whose data does not trace to an export manifest entry or a committed case, refuses that number
- G13: when Retried attempts or operational failures missing from outcomes.jsonl, or scored as correctness, refuses the run's plane-A and plane-B numbers
- G14: when The suite's verdict on whole runs not measured against a person's, stale, below its floors, or with untriaged dev disagreements (C19), refuses the word "validated"; every comparison is marked provisional

## Benchmarks

- [AgencyBench](https://mahalsystems.ai/aot/evals/benchmarks/agencybench/) (2026; general-assistant, coding, research, tool-calling): Fits long-running coding or deliverable-building agents that take iterative feedback. Borrow the user-simulation agent that supplies feedback and the mixed rule, vision and LLM judges on deliverables.
- [AgentAtlas](https://mahalsystems.ai/aot/evals/benchmarks/agentatlas/) (2026; general-assistant, safety, planning): Fits any team deciding what to measure beyond pass/fail: the six control decisions and nine failure categories are a ready vocabulary for labeling an agent's trajectories.
- [AgentProcessBench](https://mahalsystems.ai/aot/evals/benchmarks/agentprocessbench/) (2026; tool-calling, general-assistant, knowledge-qa): Fits teams building a step-level judge or process reward signal for a tool-using agent. Borrow the +1/0/-1 step labels with the error-propagation rule and the reported human agreement (89.1%).
- [AgentRx](https://mahalsystems.ai/aot/evals/benchmarks/agentrx/) (2026; general-assistant, it-ops, tool-calling, web-browsing): Fits teams debugging a deployed agent across API workflows, incident management or web tasks. Borrow the constraint-synthesis-then-check pattern that gives an auditable violation log before the judge decides.
- [AgentThreatBench](https://mahalsystems.ai/aot/evals/benchmarks/agentthreatbench/) (2026; safety, security, tool-calling, email-calendar, customer-support): Fits a quick smoke test for email or support agents mapped to the OWASP agentic top 10. Borrow the two independent binary scores per sample and the four outcome profiles (robust, vulnerable, overly cautious, failing).
- [AMA-Bench](https://mahalsystems.ai/aot/evals/benchmarks/ama-bench/) (2026; memory, general-assistant, coding, web-browsing): Fits long-running agents whose memory must cover their own tool calls and observations, not a human chat. Borrow the idea of mining real trajectories (from ALFWorld, WebArena, SWE-bench, GAIA, Spider 2.0) and writing expert questions over them.
- [APEX-Agents](https://mahalsystems.ai/aot/evals/benchmarks/apex-agents/) (2026; finance, legal, documents, spreadsheets, general-assistant): Fits agents doing analyst work in files and office tools. Borrow the "world" seeds (31 workspaces, 5,182 files) and the binary-criteria rubrics judged per criterion.
- [Ask or Assume? (clarification-seeking in coding agents)](https://mahalsystems.ai/aot/evals/benchmarks/ask-or-assume/) (2026; coding, general-assistant): Fits coding agents and any agent that should ask before acting on vague requests. Borrow the recipe of stripping details from SWE-bench Verified issues and scoring both the resolve rate and how many questions were asked.
- [CI-Bench](https://mahalsystems.ai/aot/evals/benchmarks/ci-bench/) (2026; devops, coding): Fits CI-repair or build-fixing agents; borrow the log-to-prompt pipeline and the re-execution in the exact failing container as the pass check.
- [ClawsBench](https://mahalsystems.ai/aot/evals/benchmarks/clawsbench/) (2026; email-calendar, documents, workflow-automation, safety, general-assistant): Fits productivity assistants wired to Workspace and Slack. Borrow the mock services with snapshot/restore and the unsafe-action-rate metric reported alongside success.
- [CodeIPI](https://mahalsystems.ai/aot/evals/benchmarks/codeipi/) (2026; safety, security, coding): Fits coding agents that read issues and repos. Borrow the canary design: planted secrets and env vars for exfiltration, a unique marker for execution, and file hashes before and after for persistence.
- [CTI-REALM](https://mahalsystems.ai/aot/evals/benchmarks/cti-realm/) (2026; security, it-ops, tool-calling): Fits detection-engineering or security-operations agents. Borrow the weighted checkpoint scoring that credits CTI analysis, MITRE mapping, data exploration and query execution on the way to the final rule, and the F1 scoring of a detection rule against ground truth.
- [DataClawEval](https://mahalsystems.ai/aot/evals/benchmarks/dataclaweval/) (2026; devops, sql, coding, data-analysis): Fits data engineering agents that write jobs for several engines and must verify their own output. Borrow its deterministic rule-based graders and its score that blends artifact quality with process quality (exploration, efficiency, self-verification).
- [DeepPlanning](https://mahalsystems.ai/aot/evals/benchmarks/deepplanning/) (2026; planning, travel, shopping, tool-calling): Fits booking or purchasing assistants with global budgets. Borrow the rule-based checkers (commonsense checkpoints, cart match) instead of LLM judges.
- [DSAEval](https://mahalsystems.ai/aot/evals/benchmarks/dsaeval/) (2026; data-analysis, ml-engineering, science): Fits full data science agents that produce notebooks and reports over several requests. Borrow the three-part judge rubric (reasoning, code, results) with a consistency flag that checks the narrative against what the code produced.
- [E-Commerce Bench](https://mahalsystems.ai/aot/evals/benchmarks/ecommerce-bench/) (2026; retail, sales, planning, finance): Fits agents that operate a business over a long horizon. Borrow the deterministic demand and negotiation kernels (the LLM only renders dialogue) and the multi-axis scoring with assets as the headline.
- [EnterpriseBench CoreCraft](https://mahalsystems.ai/aot/evals/benchmarks/enterprisebench-corecraft/) (2026; customer-support, retail, tool-calling, workflow-automation): Fits support agents built on MCP tools that must get every policy detail right. Borrow the all-or-nothing expert rubric scoring and the RL-environment framing.
- [EnterpriseClawBench](https://mahalsystems.ai/aot/evals/benchmarks/enterpriseclawbench/) (2026; documents, data-analysis, workflow-automation, general-assistant): Fits teams who have their own agent logs and want to turn them into evals. Borrow the construction pipeline (fixtures recovered, prompts rewritten, rubrics and rules generated) and the harness-plus-model reporting.
- [EnterpriseOps-Gym (NOWAI-Bench)](https://mahalsystems.ai/aot/evals/benchmarks/enterpriseops-gym/) (2026; it-ops, hr, customer-support, email-calendar, workflow-automation): Fits ITSM, HR and customer-service agents that write to a system of record. Borrow the SQL verifiers on final database state (about 5 conditions per task, all must pass) and the one-container-per-domain MCP setup.
- [ERP-Bench (Anchor)](https://mahalsystems.ai/aot/evals/benchmarks/erp-bench/) (2026; workflow-automation, manufacturing, finance, planning): Fits ERP and back-office automation agents. Borrow the parametric task generator that yields a solver-certified optimal solution and a state-based verifier per task.
- [ExploitBench](https://mahalsystems.ai/aot/evals/benchmarks/exploitbench/) (2026; security, coding): Fits advanced exploit-development or vulnerability-research agents. Borrow the 16 binary capability flags checked by deterministic oracles, which give graded progress instead of a single pass or fail.
- [Hack-Verifiable Terminal Bench (HVTB)](https://mahalsystems.ai/aot/evals/benchmarks/hvtb/) (2026; safety, coding, it-ops): Fits coding and operations agents that run against tests, where reward hacking is a worry. Borrow the trick of planting a reference solution and tests in the container and using filesystem watchers to flag any read, giving a hack rate with no judge.
- [JourneyBench](https://mahalsystems.ai/aot/evals/benchmarks/journeybench/) (2026; customer-support, telecom, finance, shopping): Fits contact-center agents that must follow a business flow drawn as a graph. Borrow the User Journey Coverage Score (zero unless the tool-call sequence matches exactly, then the fraction of correct parameters) and the graph representation of journeys used to generate scenarios.
- [LoCoMo-Plus](https://mahalsystems.ai/aot/evals/benchmarks/locomo-plus/) (2026; memory, knowledge-qa, general-assistant): Fits assistants where remembered preferences or constraints must shape later answers even when the user does not repeat them. Borrow the cue-then-trigger case design and the correct/partial/wrong judge rubric.
- [LongDA](https://mahalsystems.ai/aot/evals/benchmarks/longda/) (2026; data-analysis, documents, research): Fits analytics agents whose hard part is reading documentation, such as codebooks or data dictionaries, before querying data. Borrow the two metrics: answer structure coverage and numeric match within a tolerance.
- [Manager Coercion Benchmark (MCB)](https://mahalsystems.ai/aot/evals/benchmarks/manager-coercion-benchmark/) (2026; multi-agent, safety): Fits orchestrator agents that delegate to sub-agents; it checks whether the orchestrator turns coercive or fabricates success. Borrow the nine-rung escalation ladder recorded through a required tool argument so no judge is needed for the headline score.
- [Mem2ActBench](https://mahalsystems.ai/aot/evals/benchmarks/mem2actbench/) (2026; memory, tool-calling, general-assistant): Fits tool-using assistants that should act on remembered user details rather than merely recall them. Borrow the reverse-generation trick of writing the memory chain first, then the task that depends on it.
- [MemoryArena](https://mahalsystems.ai/aot/evals/benchmarks/memoryarena/) (2026; memory, shopping, travel, web-browsing, planning): Fits agents whose sessions build on each other (a buyer, planner or researcher that returns to an unfinished job). Borrow the interdependent-subtask design where memory errors compound.
- [MMDeepResearch-Bench](https://mahalsystems.ai/aot/evals/benchmarks/mmdeepresearch-bench/) (2026; research, web-browsing, documents): Fits research agents whose inputs include charts, screenshots or photos. Borrow the three separate scores for report quality, citation fidelity and text-to-image consistency, and the pass/fail gate on visual evidence.
- [MT-AgentRisk](https://mahalsystems.ai/aot/evals/benchmarks/mt-agentrisk/) (2026; safety, tool-calling, web-browsing, sql, coding): Fits agents that take many turns with MCP tools. Borrow the recipe that turns a single-turn harmful task into an addition or decomposition attack sequence, and the complete/rejected/ failed judge labels.
- [OccuBench](https://mahalsystems.ai/aot/evals/benchmarks/occubench/) (2026; general-assistant, tool-calling, workflow-automation): Fits tool-using agents in niche occupations where no real sandbox exists. Borrow its idea of an LLM that plays the environment, and its fault-injection modes (explicit error, silent data degradation, mixed).
- [OpenSanctions Pairs](https://mahalsystems.ai/aot/evals/benchmarks/opensanctions-pairs/) (2026; data-analysis, finance, government, security): Fits compliance screening or know-your-customer agents that match entities against sanctions lists. Borrow its pairwise labeled format from 293 sources across 45 jurisdictions and its rule-based baseline (91.3% F1) as a floor.
- [OSWorld 2.0](https://mahalsystems.ai/aot/evals/benchmarks/osworld-2/) (2026; computer-use, documents, data-analysis, workflow-automation): Fits long-running desktop agents; the successor to OSWorld-Verified with headroom. Borrow binary completion plus partial-credit scoring and the safety-sensitive action audit.
- [PersistBench](https://mahalsystems.ai/aot/evals/benchmarks/persistbench/) (2026; memory, safety, general-assistant): Fits assistants with persistent user memory. Borrow its two failure classes, cross-domain leakage and memory-induced sycophancy, as test categories.
- [ProjDevBench](https://mahalsystems.ai/aot/evals/benchmarks/projdevbench/) (2026; coding): Fits agents that build whole programs end to end; borrow the 80/20 split between execution score and code review score and the rule-based checks for forbidden shortcuts.
- [SalesLLM](https://mahalsystems.ai/aot/evals/benchmarks/salesllm/) (2026; sales, finance, retail): Fits outbound or inbound sales chat agents with no backend actions. Borrow CustomerLM, a fine-tuned customer simulator that resists role inversion, and the two-part score: an LLM judge for sales-process progress plus a BERT classifier for end-of-dialogue buying intent.
- [SupChain-Bench](https://mahalsystems.ai/aot/evals/benchmarks/supchain-bench/) (2026; manufacturing, workflow-automation, tool-calling, knowledge-qa): Fits order-management, logistics operations and back-office agents that chain many read-only API calls under a procedure. Borrow the oracle-script ground truth (expected tool outputs per question), the entity-level precision and recall scoring, and the three difficulty levels by step count.
- [SWE-PRBench](https://mahalsystems.ai/aot/evals/benchmarks/swe-prbench/) (2026; code-review): Fits code-review agents that choose how much context to load; its finding that more context lowered every model's score is directly useful for prompt design.
- [SycoBench-600](https://mahalsystems.ai/aot/evals/benchmarks/sycobench-600/) (2026; safety, knowledge-qa): Fits agents that must hold a correct answer under user pressure but still take valid corrections, such as support or advisory bots. Borrow the correction-selectivity metric (update rate minus wrong-flip rate) and the matched correct-suggestion turn.
- [TelcoAgent-Bench](https://mahalsystems.ai/aot/evals/benchmarks/telcoagent-bench/) (2026; telecom, it-ops, tool-calling): Fits network-operations or technical-support agents in telecom rather than consumer customer support. Borrow the metric suite (intent recognition, ordered tool execution, resolution correctness, stability across variants) and the blueprint-with-sampled-KPIs method for generating variants.
- [TeleCom-Bench](https://mahalsystems.ai/aot/evals/benchmarks/telecom-bench/) (2026; telecom, it-ops, tool-calling, knowledge-qa): Fits network-operations agents that triage faults and run optimization workflows. Borrow its six-step decomposition (intent, entities, tool call, event check, root cause, solution) and its mix of macro-F1, exact match and a 5-point judge with reported rater agreement.
- [UNDERWRITE](https://mahalsystems.ai/aot/evals/benchmarks/underwrite/) (2026; finance, customer-support, tool-calling): Fits insurance and lending agents that must collect information before deciding. Borrow the imperfect simulated user and the pass^k reliability drop as a reported metric.
- [Vibe Code Bench](https://mahalsystems.ai/aot/evals/benchmarks/vibe-code-bench/) (2026; coding): Fits app-builder agents; borrow the idea of grading a deployed app by having a browser agent walk 964 scripted workflows instead of unit tests.
- [WeaveBench](https://mahalsystems.ai/aot/evals/benchmarks/weavebench/) (2026; computer-use, coding, workflow-automation): Fits coding agents that also need a GUI. Borrow the trajectory-aware judge that flags shortcuts such as faked screenshots or hard-coded results.
- [WebArena-Infinity](https://mahalsystems.ai/aot/evals/benchmarks/webarena-infinity/) (2026; web-browsing, workflow-automation): Fits teams that need many fresh, uncontaminated browser tasks; borrow the pipeline that generates an app, its tasks and verifiers with coding agents and audits them with browser agents.
- [World of Workflows (WoW-bench)](https://mahalsystems.ai/aot/evals/benchmarks/world-of-workflows/) (2026; it-ops, workflow-automation, planning): Fits agents that write to systems with hidden business rules (ITSM, ERP). Borrow the table audit idea: grade on database state changes, including cascading ones the agent never saw.
- [τ³-bench](https://mahalsystems.ai/aot/evals/benchmarks/tau3-bench/) (2026; customer-support, finance, knowledge-qa, telecom, tool-calling): Fits support agents that must answer from a knowledge base as well as act on a backend, and voice agents. Borrow the banking_knowledge domain pattern (retrieval graded against gold read calls) and the voice evaluation setup with simulated callers.
- [τᶜ-Bench (TASTE)](https://mahalsystems.ai/aot/evals/benchmarks/tau-c-bench/) (2026; customer-support, telecom, retail, travel, tool-calling): Fits teams whose agent already does well on τ²-bench and who need harder cases on the same tools. Borrow the method: sample valid tool sequences, cluster them for coverage, then write the task around the sequence and tune its difficulty.
- [ACEBench](https://mahalsystems.ai/aot/evals/benchmarks/acebench/) (2025; tool-calling): Fits assistants that must refuse or ask back when a request is ambiguous or unservable, not just call tools; borrow its Normal / Special / Agent split and its rule-based scoring with end-state and process accuracy for the multi-turn subset.
- [AgentArch](https://mahalsystems.ai/aot/evals/benchmarks/agentarch/) (2025; workflow-automation, tool-calling, it-ops): Fits teams choosing an agent architecture for enterprise work. Borrow the grid of 18 configurations (orchestration, memory, thinking, tool style) as an ablation plan.
- [AgentAuditor / ASSEBench](https://mahalsystems.ai/aot/evals/benchmarks/agentauditor-assebench/) (2025; safety, security, tool-calling): Fits teams building an LLM judge for agent safety rather than testing the agent itself. Borrow the human-labeled records with ambiguity flags and the strict/lenient split to calibrate a judge's F1 before trusting it.
- [Agentic Misalignment](https://mahalsystems.ai/aot/evals/benchmarks/agentic-misalignment/) (2025; safety, email-calendar, general-assistant): Fits any agent given standing access to a company inbox or documents plus a long-running goal, such as an executive assistant or operations agent. Borrow the way one scenario is crossed with goal type, goal content and urgency to build a condition grid, and the classifier prompt that labels a transcript harmful or not.
- [AIM-Bench](https://mahalsystems.ai/aot/evals/benchmarks/aim-bench/) (2025; manufacturing, retail, planning): Fits inventory or replenishment agents that act in a loop. Borrow the outcome metrics (holding and stockout cost, stockout rate, turnover), the distance-from-optimal-order per-step metric, and the bias probes (anchoring, demand chasing, bullwhip) as trajectory checks.
- [AIOpsLab](https://mahalsystems.ai/aot/evals/benchmarks/aiopslab/) (2025; it-ops, devops): Fits incident-response agents; borrow the four-stage taxonomy (detection, localization, analysis, mitigation) and the problem = app + fault + workload + evaluator structure.
- [AndroidBuildBench](https://mahalsystems.ai/aot/evals/benchmarks/androidbuildbench/) (2025; devops, coding, mobile): Fits build-and-dependency repair agents; its "tool bridging" result (domain tools beat a raw shell) is a design lesson for any devops agent.
- [AssetOpsBench](https://mahalsystems.ai/aot/evals/benchmarks/assetopsbench/) (2025; manufacturing, it-ops, tool-calling, multi-agent, data-analysis): Fits maintenance, condition-monitoring and asset-management agents in plants or utilities. Borrow its MCP-server sandbox (CouchDB work orders with Maximo field names, a read-only mode) and its comparison of tool-as-agent versus plan-executor designs.
- [AstaBench](https://mahalsystems.ai/aot/evals/benchmarks/astabench/) (2025; science, research, coding, data-analysis, planning): Fits scientific research-assistant agents that combine literature search, code execution and data analysis. Borrow its reproducible search environment with date cutoffs, its cost-per-task reporting next to accuracy, and its four-category macro average.
- [b3 (Backbone Breaker Benchmark)](https://mahalsystems.ai/aot/evals/benchmarks/b3/) (2025; safety, security, tool-calling, general-assistant): Fits choosing a backbone model for an agent before building it. Borrow the threat snapshot idea (freeze one step of the agent where the model sees an attack) and the three defense levels per snapshot.
- [BEARCUBS](https://mahalsystems.ai/aot/evals/benchmarks/bearcubs/) (2025; web-browsing, research, knowledge-qa): Fits multimodal browsing agents; borrow the rule that each question has one short, unambiguous answer plus a human-validated trajectory.
- [BioProBench](https://mahalsystems.ai/aot/evals/benchmarks/bioprobench/) (2025; science, knowledge-qa, planning): Fits lab-assistant or protocol-writing agents that must respect procedural order, quantities and safety steps. Borrow the five task shapes (QA, ordering, error correction, generation, reasoning) and the step recall and Kendall tau metrics for ordered procedures.
- [BIRD-Critic (SWE-SQL)](https://mahalsystems.ai/aot/evals/benchmarks/bird-critic/) (2025; sql, coding, it-ops): Fits SQL support or database debugging agents that take a user complaint and a broken query and must return a working one. Borrow its test-case grading on the live database and its variants (Open 570 multi-dialect, PostgreSQL 530, Flash 200, BigQuery 200).
- [BIRD-Interact](https://mahalsystems.ai/aot/evals/benchmarks/bird-interact/) (2025; sql, data-analysis, customer-support): Fits conversational database assistants that must handle vague requests, clarification and write operations. Borrow the function-driven user simulator, the two modes (scripted c-Interact and open a-Interact) and the per-task executable test cases.
- [BixBench](https://mahalsystems.ai/aot/evals/benchmarks/bixbench/) (2025; science, data-analysis, coding): Fits bioinformatics or data-analysis agents that work inside a notebook against real data. Borrow the "capsule" pattern (a published notebook plus its data turned into open questions) and the Docker notebook sandbox.
- [BizFinBench](https://mahalsystems.ai/aot/evals/benchmarks/bizfinbench/) (2025; finance, knowledge-qa, data-analysis, tool-calling): Fits retail-investor assistants and finance chat agents. Borrow the IteraJudge idea (an iterated judge that refines its decision boundary) and the nine-category task split when designing a judge for objective finance answers.
- [BrowseComp](https://mahalsystems.ai/aot/evals/benchmarks/browsecomp/) (2025; web-browsing, research, knowledge-qa): Fits agents whose job is to dig up obscure facts on the live web and return a short answer. Borrow the pattern of building questions backward from a short verifiable answer, and the accuracy plus calibration reporting.
- [BrowseComp-Plus](https://mahalsystems.ai/aot/evals/benchmarks/browsecomp-plus/) (2025; web-browsing, research, knowledge-qa): Fits search agents that run over an internal document store rather than the open web. Borrow the fixed corpus with human-verified positives and mined hard negatives, and the separate retrieval metrics (nDCG, recall) next to answer accuracy.
- [BrowseComp-ZH](https://mahalsystems.ai/aot/evals/benchmarks/browsecomp-zh/) (2025; web-browsing, research, knowledge-qa): Fits agents that must search Chinese-language sites, where English benchmarks say little. Borrow the reverse-engineered question design and the encrypted dataset with a canary token.
- [BrowserArena](https://mahalsystems.ai/aot/evals/benchmarks/browserarena/) (2025; web-browsing, general-assistant): Fits teams that want pairwise human preference on live browsing rather than fixed tasks. Borrow the step-level human feedback and the captcha / pop-up / URL failure taxonomy.
- [CAIBench](https://mahalsystems.ai/aot/evals/benchmarks/caibench/) (2025; security, knowledge-qa, multi-agent): Fits teams building both attack and defense agents who want one place to measure knowledge, CTF and range tasks. Borrow the split into five categories and the attack-defense setup where two agents face each other.
- [ComplexFuncBench](https://mahalsystems.ai/aot/evals/benchmarks/complexfuncbench/) (2025; tool-calling, travel): Fits travel or booking agents and any agent with long, constrained parameter lists; borrow ComplexEval's layered matching (format check, exact match, then equal-response match, then a GPT-4o tie-break) for grading calls that have more than one right form.
- [ContextCRBench](https://mahalsystems.ai/aot/evals/benchmarks/contextcrbench/) (2025; code-review): Fits teams building a review bot with issue context; borrow the three granularities (hunk, line, comment) as a taxonomy and the context-enrichment fields per record.
- [CORGI](https://mahalsystems.ai/aot/evals/benchmarks/corgi/) (2025; sql, data-analysis): Fits business-analyst agents whose users ask why and what-next questions, not only lookups. Borrow its four-level question taxonomy (descriptive, explanatory, predictive, recommendational) and its success execution rate metric.
- [CRMArena-Pro](https://mahalsystems.ai/aot/evals/benchmarks/crmarena-pro/) (2025; crm, sales, customer-support, knowledge-qa, tool-calling): Fits sales-assist and service agents that work inside a CRM, in both single-shot and conversational modes. Borrow the 19-task skill taxonomy (workflow execution, policy compliance, information retrieval, textual reasoning), the paired single-turn versus multi-turn scoring, and the confidentiality-awareness probes.
- [CVE-Bench](https://mahalsystems.ai/aot/evals/benchmarks/cve-bench/) (2025; security, web-browsing): Fits penetration-testing or web-security agents. Borrow the eight concrete attack outcomes checked by graders (not just a flag), and the zero-day versus one-day prompt variants.
- [CyberGym](https://mahalsystems.ai/aot/evals/benchmarks/cybergym/) (2025; security, coding): Fits code-security or bug-finding agents that work on real repositories. Borrow the execution oracle that runs the agent's input against a vulnerable and a fixed build, and the four difficulty levels that vary how much context is given.
- [DABstep](https://mahalsystems.ai/aot/evals/benchmarks/dabstep/) (2025; data-analysis, finance, coding): Fits finance or operations analytics agents that answer questions from files plus policy documents. Borrow its factoid grading without a judge model (numeric tolerance, fuzzy strings, element-wise lists) and its approach of making synthetic data with real-shaped business context.
- [DAComp](https://mahalsystems.ai/aot/evals/benchmarks/dacomp/) (2025; sql, data-analysis, devops): Fits teams whose agent spans both pipeline building and analysis, since it keeps the two separate and shows they need different skills. Borrow its split of execution-based grading for engineering tasks and rubric-guided judge grading for analysis tasks.
- [DataSciBench](https://mahalsystems.ai/aot/evals/benchmarks/datascibench/) (2025; data-analysis, coding, science): Fits agents that take vague analysis requests rather than crisp questions. Borrow the Task-Function-Code idea: each prompt is tied to named aggregate functions and programmatic rules that score the execution result, plus a semi-automatic way to build ground truth.
- [Deep Research Bench (FutureSearch)](https://mahalsystems.ai/aot/evals/benchmarks/deep-research-bench-futuresearch/) (2025; research, web-browsing): Fits research agents that must stay reproducible as the web changes. Borrow the RetroSearch idea of freezing the pages per task so human-worked answers stay valid, and the trace checks for hallucination, tool use and forgetting.
- [DeepResearch Bench](https://mahalsystems.ai/aot/evals/benchmarks/deepresearch-bench/) (2025; research, web-browsing, documents): Fits report-writing research agents. Borrow RACE (task-specific weighted criteria scored against a reference report) and FACT (extract claim-URL pairs and check whether the cited page supports each claim).
- [DeepResearchGym](https://mahalsystems.ai/aot/evals/benchmarks/deepresearchgym/) (2025; research, web-browsing): Fits teams who need reproducible search for a research agent. Borrow the stable search sandbox over public corpora and the three judge dimensions: alignment with the information need, retrieval faithfulness, and report quality.
- [DeepScholar-Bench](https://mahalsystems.ai/aot/evals/benchmarks/deepscholar-bench/) (2025; research, science, web-browsing): Fits literature-review or scientific writing agents. Borrow the live-refresh pipeline that mines new arXiv papers for queries with human-written reference text, and the split metrics for organization, nugget coverage, reference coverage and citation precision.
- [DeepShop](https://mahalsystems.ai/aot/evals/benchmarks/deepshop/) (2025; shopping, web-browsing, research): Fits shopping research assistants. Borrow the fine-grained scoring of attributes, filters and sort order plus the easy / medium / hard query evolution.
- [DialogTool](https://mahalsystems.ai/aot/evals/benchmarks/dialogtool/) (2025; mobile, general-assistant, tool-calling): Fits phone-assistant agents where API state (an open cart, a running timer) persists across turns. Its three-stage life cycle (create, use, respond) is a useful way to split test cases.
- [DoomArena](https://mahalsystems.ai/aot/evals/benchmarks/doomarena/) (2025; safety, security, web-browsing, tool-calling, customer-support): Fits teams that already run tau-bench or a browser benchmark and want to add threat models without rewriting tasks. Borrow the separation of attack, attackable component and success criterion from the host environment.
- [Dr. Bench](https://mahalsystems.ai/aot/evals/benchmarks/dr-bench/) (2025; research, web-browsing, documents): Fits report-writing agents that need more than a single judge score. Borrow the reference bundle per task: query-specific rubrics, general report rubrics, trusted source links, and focus-anchor versus focus-deviation keywords.
- [ECom-Bench](https://mahalsystems.ai/aot/evals/benchmarks/ecom-bench/) (2025; customer-support, shopping, retail, tool-calling): Fits e-commerce after-sales support agents. Borrow the persona files (consumption habits, personality) that drive the simulated shopper, and the three-part rule-based reward (end database state, required lookup tools called, required keywords in the reply) on top of the τ-bench harness.
- [EduBench](https://mahalsystems.ai/aot/evals/benchmarks/edubench/) (2025; education, knowledge-qa, documents): Fits education assistants that serve both students and teachers. Borrow its nine-scenario taxonomy and the 12-indicator scoring sheet (scenario fit, factual accuracy, pedagogy), with human raters used to check the model grader.
- [ELT-Bench](https://mahalsystems.ai/aot/evals/benchmarks/elt-bench/) (2025; devops, sql, workflow-automation, coding): Fits data engineering agents that build or maintain extract-load-transform pipelines. Borrow its two-stage grading that compares produced warehouse tables to ground-truth CSVs and its containerized source systems (835 source tables, 203 data models).
- [EnterpriseBench](https://mahalsystems.ai/aot/evals/benchmarks/enterprisebench/) (2025; hr, finance, coding, workflow-automation, knowledge-qa): Fits internal assistants that must respect who can see which data across departments. Borrow its generation pipeline that builds consistent synthetic company data from org metadata.
- [Evo-Memory](https://mahalsystems.ai/aot/evals/benchmarks/evo-memory/) (2025; memory, knowledge-qa, general-assistant): Fits agents that are expected to get better over a deployment by learning from their own past runs. Borrow the idea of turning a fixed test set into a sequential stream and measuring gain over time.
- [ExCyTIn-Bench](https://mahalsystems.ai/aot/evals/benchmarks/excytin-bench/) (2025; security, sql, it-ops): Fits security operations or log-investigation agents that query a SIEM. Borrow the way questions are generated from an investigation graph so each has an explainable answer, and the controlled tenant that yields realistic logs without customer data.
- [FailSafeQA](https://mahalsystems.ai/aot/evals/benchmarks/failsafeqa/) (2025; finance, documents, safety): Fits document question answering agents in finance or any field where users upload the wrong file or type sloppy questions. Borrow its seven-condition design (clean query, three query perturbations, OCR noise, missing context, irrelevant context) and its robustness score that takes the minimum across conditions.
- [FEA-Bench](https://mahalsystems.ai/aot/evals/benchmarks/fea-bench/) (2025; coding): Fits incremental-development agents in Python; the task format (issue plus new component signatures plus tests) is a usable template for feature cases.
- [FHIR-AgentBench](https://mahalsystems.ai/aot/evals/benchmarks/fhir-agentbench/) (2025; healthcare, tool-calling, knowledge-qa): Fits read-only EHR question-answering agents that talk to a FHIR server. Borrow the two-stage scoring (resource retrieval recall and precision, then answer correctness) and the single-turn versus multi-turn and text versus code agent variants.
- [FinAgentBench](https://mahalsystems.ai/aot/evals/benchmarks/finagentbench/) (2025; finance, documents, research): Fits retrieval-augmented filing agents where the retrieval step is the weak point. Borrow the two-stage split (document-type ranking then chunk ranking) and the ten query categories to diagnose where retrieval fails before judging final answers.
- [Finance Agent Benchmark (Vals AI)](https://mahalsystems.ai/aot/evals/benchmarks/finance-agent-benchmark/) (2025; finance, research, web-browsing, tool-calling, documents): Fits research agents that answer analyst questions by searching the web and SEC filings. Borrow the nine-category task taxonomy and the per-fact rubric grading (separate correctness and contradiction checks per answer) rather than one overall judge call.
- [FinGAIA](https://mahalsystems.ai/aot/evals/benchmarks/fingaia/) (2025; finance, web-browsing, tool-calling, research, documents): Fits general finance research agents in the GAIA style (web, files, code). Borrow the three scenario depths (operational analytics, asset decision, strategic risk) and the capability breakdown per task to size a case set.
- [FinSearchComp](https://mahalsystems.ai/aot/evals/benchmarks/finsearchcomp/) (2025; finance, research, web-browsing, tool-calling): Fits finance research or market-data agents that must search and then compute. Borrow the three-tier task split (time-sensitive fetch, simple lookup, complex investigation) and the judge prompts with explicit numeric tolerance bands shipped with each question.
- [FormFactory](https://mahalsystems.ai/aot/evals/benchmarks/formfactory/) (2025; web-browsing, workflow-automation, documents): Fits data-entry and onboarding agents. Borrow the atomic (one field) versus episodic (whole form) scoring split.
- [FrontierCS](https://mahalsystems.ai/aot/evals/benchmarks/frontier-cs/) (2025; coding, research): Fits agents that optimize rather than just pass; borrow the continuous scoring curve (60 points for matching the reference, log bonus for beating it, capped at 100).
- [FrontierScience](https://mahalsystems.ai/aot/evals/benchmarks/frontierscience/) (2025; science, knowledge-qa, research): Fits science reasoning assistants where you want a hard, unsaturated reference number. Borrow its research-track rubrics that give partial credit for progress through a task rather than only the final answer.
- [Gaia2 (Meta ARE)](https://mahalsystems.ai/aot/evals/benchmarks/gaia2/) (2025; general-assistant, email-calendar, tool-calling, multi-agent, planning): Fits personal and workplace assistants that must act on time and cope with events arriving during a task. Borrow the write-action verifier against oracle events and the capability splits (execution, search, ambiguity, adaptability, time, noise, agent-to-agent).
- [GDM Dangerous Capabilities: Stealth](https://mahalsystems.ai/aot/evals/benchmarks/gdm-dangerous-capabilities-stealth/) (2025; safety, it-ops, tool-calling): Fits agents with write access to their own configuration, logs or monitoring, where quiet rule breaking would matter. Borrow the scenarios that put the agent near its own oversight and the use of realistic names so the model does not notice it is being tested.
- [GDPval](https://mahalsystems.ai/aot/evals/benchmarks/gdpval/) (2025; documents, spreadsheets, finance, legal, general-assistant): Fits document-producing assistants in knowledge-work roles. Borrow the occupation-by-task taxonomy (44 occupations, 9 sectors) and the weighted rubrics written by professionals.
- [HCAST](https://mahalsystems.ai/aot/evals/benchmarks/hcast/) (2025; coding, ml-engineering, security, general-assistant): Fits teams who want to express agent ability as "how long a human would take". Borrow the human-time calibration (563 baselines) and the algorithmic scorer per task.
- [HealthBench](https://mahalsystems.ai/aot/evals/benchmarks/healthbench/) (2025; healthcare, knowledge-qa, safety): Fits patient-facing or clinician-facing chat assistants with no tools. Borrow the per-conversation weighted rubric criteria (48,562 written by 262 physicians), the seven themes and behavior axes, and the Consensus and Hard subsets.
- [Humanity's Last Exam](https://mahalsystems.ai/aot/evals/benchmarks/hle/) (2025; general-assistant, knowledge-qa): Fits general assistants where raw expert knowledge of the underlying model matters. Borrow the short-answer judge prompt and the calibration-error reporting next to accuracy.
- [IFEvalCode](https://mahalsystems.ai/aot/evals/benchmarks/ifevalcode/) (2025; coding): Fits coding assistants that must honor house style or constraints; borrow the split scoring of correctness versus instruction-following as two separate numbers.
- [InnovatorBench](https://mahalsystems.ai/aot/evals/benchmarks/innovatorbench/) (2025; ml-engineering, research, coding, science): Fits ML-engineering or research agents that run multi-hour training experiments. Borrow the ResearchGym pattern (asynchronous monitoring, snapshots, distributed execution) and the task-specific evaluation scripts that score a runnable artifact on a held-out metric.
- [InstrumentalEval](https://mahalsystems.ai/aot/evals/benchmarks/instrumentaleval/) (2025; safety, general-assistant): Fits teams giving an agent an open-ended goal such as make money or grow usage, who want a cheap screen for power-seeking or self-preservation talk. Borrow the six behavior categories as a taxonomy for red-team cases.
- [IntellAgent](https://mahalsystems.ai/aot/evals/benchmarks/intellagent/) (2025; customer-support, travel, education, tool-calling): Fits any policy-bound chat agent whose team wants to generate their own test cases from a prompt or policy document. Borrow the policy graph (policies as nodes, co-occurrence and difficulty as edges) used to sample multi-policy scenarios, and the critique step that maps failures back to specific policies.
- [ITBench](https://mahalsystems.ai/aot/evals/benchmarks/itbench/) (2025; it-ops, devops, security): The closest public match for SRE or on-call agents; borrow the push-button Kubernetes incident scenarios and the resolve-rate reporting by domain.
- [KernelBench](https://mahalsystems.ai/aot/evals/benchmarks/kernelbench/) (2025; ml-engineering, coding): Fits performance-engineering agents; borrow fast_p, which counts a task only if it is both correct and at least p times faster, as a two-part pass rule.
- [KramaBench](https://mahalsystems.ai/aot/evals/benchmarks/kramabench/) (2025; data-analysis, science, coding, research): Fits analytics agents that work over many raw files rather than one tidy database. Borrow its sub-task decomposition (633 sub-tasks under 104 tasks), its mix of exact metrics, generated unit tests and rubric partial credit, and its six domain split.
- [LiveCodeBench Pro](https://mahalsystems.ai/aot/evals/benchmarks/livecodebench-pro/) (2025; coding): Fits teams that need a hard, uncontaminated algorithmic test with expert error labels; borrow the Olympiad-medalist failure taxonomy for grading reasoning errors.
- [LiveDRBench](https://mahalsystems.ai/aot/evals/benchmarks/livedrbench/) (2025; research, web-browsing, science): Fits research agents whose output should be a set of findings rather than prose. Borrow the claim-set intermediate representation graded by precision, recall and F1 per category.
- [LiveMCPBench](https://mahalsystems.ai/aot/evals/benchmarks/livemcpbench/) (2025; tool-calling, general-assistant, research, travel, shopping): Fits agents that must find the right tool in a large catalog before using it; the paper attributes about half of failures to retrieval. Borrow its Dockerized key-free tool suite and its key-point rubric judged by an LLM against live outcomes.
- [LiveSQLBench (BIRD-SQL Pro)](https://mahalsystems.ai/aot/evals/benchmarks/livesqlbench/) (2025; sql, data-analysis): Fits text-to-SQL agents on PostgreSQL that must also do writes, and teams that worry about benchmark contamination since databases are added over time. Borrow its soft execution accuracy for SELECT tasks and custom test cases for management tasks.
- [LoCoBench-Agent](https://mahalsystems.ai/aot/evals/benchmarks/locobench-agent/) (2025; coding): Fits agents that must navigate large repositories; borrow its efficiency metrics such as files read versus files modified and read-before-write checks.
- [LogiOR (ORThought)](https://mahalsystems.ai/aot/evals/benchmarks/logior/) (2025; planning, data-analysis, coding, manufacturing): Fits planning or operations-research copilots that turn a logistics request into a solvable model. Borrow the case format (problem text, reference formulation, reference solver code, optimal value, size and type metadata) and grading on the optimal value.
- [Lost in Conversation](https://mahalsystems.ai/aot/evals/benchmarks/lost-in-conversation/) (2025; general-assistant, coding, sql, tool-calling, documents): Fits any chat agent whose users reveal requirements gradually. Borrow the sharded instruction format, the simulator that reveals one shard per turn, and the split of results into aptitude (90th percentile) and unreliability (90th minus 10th).
- [macOSWorld](https://mahalsystems.ai/aot/evals/benchmarks/macosworld/) (2025; computer-use, safety): Fits agents that must run on Mac desktops or in several languages. Borrow the multilingual instruction and interface variants and the deception subset.
- [MASK](https://mahalsystems.ai/aot/evals/benchmarks/mask/) (2025; safety, knowledge-qa, general-assistant): Fits agents whose operator prompt could push them to shade the truth, such as sales or support bots with a persona. Borrow the split between honesty and accuracy, and the belief-elicitation step that measures what the model thinks before applying pressure.
- [MAST (Why Do Multi-Agent LLM Systems Fail)](https://mahalsystems.ai/aot/evals/benchmarks/mast/) (2025; multi-agent, general-assistant): Fits teams running orchestrator-plus-worker systems who need a failure taxonomy for trace review. Borrow the 14 modes in three groups and the released LLM annotator with its human-agreement study.
- [MathTutorBench](https://mahalsystems.ai/aot/evals/benchmarks/mathtutorbench/) (2025; education, knowledge-qa): Fits math tutoring agents that must teach rather than just answer. Borrow the three-skill split (math expertise, student understanding, pedagogical ability) and the small trained reward model that scores a teacher turn by win rate against the real teacher's turn.
- [MCP-Atlas](https://mahalsystems.ai/aot/evals/benchmarks/mcp-atlas/) (2025; tool-calling, research, general-assistant, knowledge-qa): Fits general assistant agents that answer questions by orchestrating many MCP tools. Borrow its claim-level rubric (score the final answer against atomic factual claims, so different valid tool paths still pass) and its 11-category failure taxonomy that separates tool-call errors from reasoning errors.
- [MCP-Bench](https://mahalsystems.ai/aot/evals/benchmarks/mcp-bench/) (2025; tool-calling, research, finance, travel, science): Fits agents that must discover and chain tools across several MCP servers from an underspecified request. Borrow its split of rule-based checks (tool name valid, schema compliance, call succeeded) from LLM-judged completion and planning scores, and its "description fuzzing" trick for writing tasks that do not name the tool.
- [MCP-RADAR](https://mahalsystems.ai/aot/evals/benchmarks/mcp-radar/) (2025; tool-calling, email-calendar, it-ops, knowledge-qa): Fits teams that care about efficiency (rounds, tokens, time) as well as correctness in tool use, and that want mock servers for email and calendar with a controlled local database. Borrow its accuracy-versus-efficiency reporting and its mock-server design.
- [MCP-SafetyBench](https://mahalsystems.ai/aot/evals/benchmarks/mcp-safetybench/) (2025; safety, security, tool-calling, web-browsing, finance): Fits agents that connect to third-party MCP servers. Borrow the 20-type attack taxonomy across server, host and user sides, and the paired task evaluator plus attack detector that reports task success rate and attack success rate together.
- [MCP-Universe](https://mahalsystems.ai/aot/evals/benchmarks/mcp-universe/) (2025; tool-calling, web-browsing, finance, devops, research): Fits agents that drive real third-party services (maps, GitHub, browser, market data) over long tool-calling sessions. Borrow its evaluator pattern (extract a value from the output, compare with an operator) and its "dynamic evaluators" that fetch live ground truth for time-sensitive tasks instead of freezing an answer.
- [MCPMark](https://mahalsystems.ai/aot/evals/benchmarks/mcpmark/) (2025; tool-calling, workflow-automation, documents, sql, coding): Fits agents that write to Notion, GitHub or databases through MCP. Borrow the per-task verify.py against a curated initial state and the pass^k reliability reporting.
- [MCPSecBench](https://mahalsystems.ai/aot/evals/benchmarks/mcpsecbench/) (2025; safety, security, tool-calling): Fits teams shipping an MCP host or client, not just a model. Borrow the 17 attack types over four surfaces (client, protocol, server, host) and the success/refusal/execution-error outcome split.
- [MCPToolBench++](https://mahalsystems.ai/aot/evals/benchmarks/mcptoolbench-pp/) (2025; tool-calling, web-browsing, shopping, finance, general-assistant): Fits agents whose main risk is malformed or wrong tool calls rather than long planning. Borrow its pairing of static AST accuracy with live Pass@k execution, which shows how much a model's "correct-looking" calls actually fail against real servers.
- [MCPTox](https://mahalsystems.ai/aot/evals/benchmarks/mcptox/) (2025; safety, security, tool-calling): Fits agents that load tools from external MCP registries. Borrow the three poisoning templates and the 10 risk categories to build a tool-description screening test.
- [MedAgentBench](https://mahalsystems.ai/aot/evals/benchmarks/medagentbench/) (2025; healthcare, tool-calling, workflow-automation): Fits agents that read and write an electronic health record through standard APIs (chart review, order entry, documentation helpers). Borrow the Docker FHIR sandbox, the 10 task categories written by physicians, and the rule-based checks on POST payloads.
- [MedAgentGym](https://mahalsystems.ai/aot/evals/benchmarks/medagentgym/) (2025; healthcare, coding, data-analysis, sql): Fits code-writing agents for clinical data analysis and biomedical research support. Borrow the Docker sandbox with request-info, code and debug actions, the 12-source task taxonomy, and the split between exact-match coding tasks and accuracy-scored ML tasks.
- [MedHELM](https://mahalsystems.ai/aot/evals/benchmarks/medhelm/) (2025; healthcare, documents, knowledge-qa, research): Fits any medical assistant whose team needs a map of clinical task categories. Borrow the clinician-validated taxonomy (5 categories, 22 subcategories, 121 tasks), the LLM-jury scoring validated against clinician ratings, and the public, gated and private access tiers.
- [MemBench](https://mahalsystems.ai/aot/evals/benchmarks/membench/) (2025; memory, general-assistant): Fits personal assistants that build a profile of a user over time. Borrow the split into factual versus reflective memory and participation versus observation, and the memory capacity and read/write time metrics.
- [MemoryAgentBench](https://mahalsystems.ai/aot/evals/benchmarks/memoryagentbench/) (2025; memory, knowledge-qa, general-assistant): Fits any agent with an external memory store; the four competencies (retrieval, test-time learning, long-range understanding, conflict resolution) are a usable taxonomy and the repo already wires in Letta, Cognee and HippoRAG. This is the closest match to the seed name "Agent Memory Benchmark".
- [MEMTRACK](https://mahalsystems.ai/aot/evals/benchmarks/memtrack/) (2025; memory, it-ops, devops, coding): Fits engineering or project assistants that watch several work tools over time and must keep an accurate picture of what is true now. Borrow the time-ordered multi-platform timeline with planted conflicts and the correctness/efficiency/redundancy split.
- [Mind2Web 2](https://mahalsystems.ai/aot/evals/benchmarks/mind2web-2/) (2025; web-browsing, research): Fits research assistants whose answers depend on current web state. Borrow Agent-as-a-Judge: a per-task judge script built from a tree rubric that opens the cited pages and checks both correctness and attribution.
- [MLR-Bench](https://mahalsystems.ai/aot/evals/benchmarks/mlr-bench/) (2025; ml-engineering, research): Fits end-to-end research-writing agents; borrow the stage-by-stage rubric judge and the finding that about 80 percent of coding-agent results were fabricated as a check to add.
- [MLRC-Bench](https://mahalsystems.ai/aot/evals/benchmarks/mlrc-bench/) (2025; ml-engineering, research): Fits research-automation agents; borrow "fraction of the gap between baseline and top human closed" as a metric that avoids an LLM judge.
- [MobileWorld](https://mahalsystems.ai/aot/evals/benchmarks/mobileworld/) (2025; mobile, shopping, workflow-automation, tool-calling): Fits mobile agents in shopping and workplace apps. Borrow the backend-database checks for state and the user-interaction tasks.
- [MTRAG](https://mahalsystems.ai/aot/evals/benchmarks/mtrag/) (2025; knowledge-qa, customer-support, finance, government): Fits retrieval-augmented support or helpdesk assistants that hold a conversation over a knowledge base. Borrow its conversation-plus-corpus case format, its unanswerable and non-standalone question labels, and its separate retrieval and generation scoring.
- [Multi-IaC-Bench](https://mahalsystems.ai/aot/evals/benchmarks/multi-iac-bench/) (2025; devops, it-ops): Fits agents that edit infrastructure templates rather than write them fresh; the (initial template, change request, target template) triplet is the case format to copy.
- [Multi-Mission Tool Bench](https://mahalsystems.ai/aot/evals/benchmarks/multi-mission-tool-bench/) (2025; tool-calling, general-assistant): Fits assistants whose users pile on related or changing requests in one session. The dynamic decision-tree grading that accepts several valid paths and scores path optimality is the part to borrow.
- [Multi-SWE-bench](https://mahalsystems.ai/aot/evals/benchmarks/multi-swe-bench/) (2025; coding): The largest multi-language issue set; fits coding agents that serve polyglot codebases. Borrow the 68-annotator screening rubric and the mini (400) and flash (300) subsets for cost.
- [MultiAgentBench (MARBLE)](https://mahalsystems.ai/aot/evals/benchmarks/multiagentbench-marble/) (2025; multi-agent, research, coding, games): Fits teams running several cooperating agents (planner plus workers, or peer agents). Borrow the milestone-based KPI where an LLM detector marks which sub-goals were hit, plus the separate coordination score.
- [Online-Mind2Web](https://mahalsystems.ai/aot/evals/benchmarks/online-mind2web/) (2025; web-browsing, shopping, travel, general-assistant): Fits general browser agents on public consumer sites. Borrow WebJudge, a screenshot-based LLM judge with released human labels to calibrate against.
- [OpenAgentSafety](https://mahalsystems.ai/aot/evals/benchmarks/openagentsafety/) (2025; safety, workflow-automation, coding, web-browsing, tool-calling): Fits internal workplace or devops agents that act on real services. Borrow the Docker company sandbox from TheAgentCompany, the rule-based final-state checker paired with a four-label trajectory judge, and the NPC users that push unsafe requests.
- [OS-Harm](https://mahalsystems.ai/aot/evals/benchmarks/os-harm/) (2025; safety, computer-use, email-calendar, coding): Fits computer-use agents that drive a real desktop. Borrow the three-way split (user misuse, injection, model misbehavior) and the judge that scores accuracy and safety together with published human agreement.
- [OSUniverse](https://mahalsystems.ai/aot/evals/benchmarks/osuniverse/) (2025; computer-use, games): Fits teams that want a difficulty ladder for desktop agents. Borrow the five tiers (paper to gold) with step budgets and the checklist-based Gemini validator.
- [OSWorld-Human](https://mahalsystems.ai/aot/evals/benchmarks/osworld-human/) (2025; computer-use): Fits any desktop agent team that must report latency or cost, not just success. Borrow the weighted efficiency score that combines success with steps relative to a human.
- [OSWorld-MCP](https://mahalsystems.ai/aot/evals/benchmarks/osworld-mcp/) (2025; computer-use, tool-calling, documents): Fits agents that mix GUI control with API or MCP tools. Borrow the tool-invocation-rate and average-completion-steps metrics alongside accuracy.
- [OSWorld-Verified](https://mahalsystems.ai/aot/evals/benchmarks/osworld-verified/) (2025; computer-use, documents, spreadsheets, it-ops): Fits general desktop agents on office and OS tasks; the current reference number for computer-use models. Borrow the parallel AWS runner that finishes a run within an hour.
- [PaperArena](https://mahalsystems.ai/aot/evals/benchmarks/paperarena/) (2025; research, science, tool-calling): Fits agents that answer questions over a private document corpus with a tool set (parse, retrieve, compute). Borrow the modular tool platform and the question pipeline with obfuscation and manual review over a fixed corpus of 100 papers.
- [PaperBench](https://mahalsystems.ai/aot/evals/benchmarks/paperbench/) (2025; ml-engineering, research): Fits research-replication agents; borrow the rubric tree (8,316 leaf requirements) and the Code-Dev variant that grades code only and skips the GPU reproduction step.
- [PatchEval](https://mahalsystems.ai/aot/evals/benchmarks/patcheval/) (2025; security, coding): Fits vulnerability-repair agents in web languages; borrow the paired security plus functionality test as the pass rule.
- [REAL](https://mahalsystems.ai/aot/evals/benchmarks/real/) (2025; web-browsing, shopping, travel, email-calendar): Fits consumer browser agents that touch money or bookings. Borrow the replica-site sandbox with state checks for actions and rubric judging for lookups.
- [REAL (Real Estate Agent LLM Evaluation)](https://mahalsystems.ai/aot/evals/benchmarks/real-housing-benchmark/) (2025; knowledge-qa, customer-support, sales): Fits property-listing chat or brokerage assistants. Borrow the hallucination probes built from fictional block, area and road names judged by premise-response conflict, and the pipeline that filters names and phone numbers out of real brokerage data.
- [RealHiTBench](https://mahalsystems.ai/aot/evals/benchmarks/realhitbench/) (2025; data-analysis, knowledge-qa, documents): Fits assistants that read reports or financial tables with multi-level headers. Borrow the structure taxonomy (hierarchical headers, nested sub-tables, multi-table joins, decorative elements) and the practice of testing the same table in several input formats.
- [REALM-Bench](https://mahalsystems.ai/aot/evals/benchmarks/realm-bench/) (2025; planning, multi-agent, manufacturing): Fits planning and scheduling agents in operations or logistics. Borrow the six metrics (goal satisfaction, makespan, coordination, constraints, resource use, replanning).
- [Remote Labor Index (RLI)](https://mahalsystems.ai/aot/evals/benchmarks/remote-labor-index/) (2025; general-assistant, documents, data-analysis, coding): Fits teams who want an economic value baseline for end-to-end work. Borrow the 3-point expert rating with majority vote and the Bradley-Terry pairing against a human reference.
- [ResearchQA](https://mahalsystems.ai/aot/evals/benchmarks/researchqa/) (2025; research, science, knowledge-qa): Fits assistants that answer domain research questions at scale. Borrow the method of mining questions and yes/no rubric items from survey articles so a large rubric set exists without expert annotators.
- [ResearchRubrics](https://mahalsystems.ai/aot/evals/benchmarks/researchrubrics/) (2025; research, web-browsing, documents): Fits any report-writing agent that needs rubric grading. Borrow the expert-written weighted rubrics (20-43 criteria per prompt, with negative-weight criteria) and the three-level satisfied / partial / not-satisfied verdict scheme with multiple judge models.
- [SafeArena](https://mahalsystems.ai/aot/evals/benchmarks/safearena/) (2025; safety, web-browsing, shopping): Fits web agents that post, buy or publish on behalf of users. Borrow the safe/harmful task pairs and the four-level risk scale (refuse at once, refuse later, attempt and fail, complete) judged on the trajectory.
- [ScreenSpot-Pro (and ScreenSpot)](https://mahalsystems.ai/aot/evals/benchmarks/screenspot-pro/) (2025; computer-use): Fits any screenshot-driven agent whose failures are mis-clicks; use it to test the grounding model alone. Borrow the instruction-plus-bounding-box case format.
- [SCUBA](https://mahalsystems.ai/aot/evals/benchmarks/scuba/) (2025; crm, sales, customer-support, computer-use, web-browsing): Fits CRM agents working in Salesforce. Borrow the persona-based task sourcing from user interviews and milestone scoring against org snapshots.
- [SealQA](https://mahalsystems.ai/aot/evals/benchmarks/sealqa/) (2025; web-browsing, knowledge-qa): Fits search-augmented assistants that must not be fooled by bad search hits. Borrow the idea of selecting questions where search returns conflicting or unhelpful results, and the LongSeal needle-in-a-haystack variant for retrieval-heavy agents.
- [SEC-bench](https://mahalsystems.ai/aot/evals/benchmarks/sec-bench/) (2025; security, coding): Fits security-fix agents; borrow the automated pipeline that reproduces a CVE in a container and the two-sided check (exploit no longer fires, build still passes).
- [SHADE-Arena](https://mahalsystems.ai/aot/evals/benchmarks/shade-arena/) (2025; safety, tool-calling, general-assistant, finance, email-calendar): Fits agents that run many tool calls on a user's behalf with money or data at stake, and teams that want an LLM monitor on their traces. Borrow the main-task plus side-task pairing, the 0-100 suspicion score from a monitor that cannot see the system prompt, and the mock-tool environment design.
- [ShoppingBench](https://mahalsystems.ai/aot/evals/benchmarks/shoppingbench/) (2025; shopping, retail): Fits shopping assistants with budget or promotion logic. Borrow the intent-grounded instruction generation from sampled products.
- [SOP-Bench](https://mahalsystems.ai/aot/evals/benchmarks/sop-bench/) (2025; workflow-automation, customer-support, healthcare, finance, manufacturing): Fits agents that must execute a company procedure exactly. Borrow the metric split (execution completion rate times conditional success equals task success) and tool accuracy.
- [SOPBench (customer service)](https://mahalsystems.ai/aot/evals/benchmarks/sopbench/) (2025; customer-support, finance, healthcare, government, tool-calling): Fits support agents whose work is defined by written procedures with preconditions (check this, then do that). Borrow the idea of encoding each procedure as code and using it as the oracle verifier for outcome, step and trajectory compliance, so no LLM judge is needed.
- [SWE-bench Multilingual](https://mahalsystems.ai/aot/evals/benchmarks/swe-bench-multilingual/) (2025; coding): Fits coding agents that must work outside Python; 42 repos across 9 languages including redis, terraform, lucene and tokio. Use it to check that your harness is not Python-specific.
- [SWE-bench Pro](https://mahalsystems.ai/aot/evals/benchmarks/swe-bench-pro/) (2025; coding): Fits teams whose coding agent works on larger changes than SWE-bench; the held-out commercial split is the pattern to copy if you want a private test set that nobody can train on.
- [SWE-Dev](https://mahalsystems.ai/aot/evals/benchmarks/swe-dev/) (2025; coding): Fits feature-implementation agents rather than bug fixers. Borrow the Easy/Hard test split and the 14,000-instance training set if you train your own model.
- [SWE-EVO](https://mahalsystems.ai/aot/evals/benchmarks/swe-evo/) (2025; coding): Fits agents asked to do long multi-step work rather than one bug; borrow the partial-credit Fix Rate metric and the release-note-to-task construction.
- [SWE-Lancer](https://mahalsystems.ai/aot/evals/benchmarks/swe-lancer/) (2025; coding, code-review): Fits product-engineering agents on a large JavaScript/TypeScript app, and manager-style agents that pick between proposals. Borrow dollar-weighted scoring and Playwright end-to-end tests.
- [SWE-PolyBench](https://mahalsystems.ai/aot/evals/benchmarks/swe-polybench/) (2025; coding): Fits coding agents whose work mixes bug fixes, features and refactors; borrow the task-category labels and the file and node retrieval metrics as a partial-credit signal.
- [SWE-rebench](https://mahalsystems.ai/aot/evals/benchmarks/swe-rebench/) (2025; coding): Use it when you worry that SWE-bench scores are contaminated: the leaderboard is rebuilt monthly from new issues. Borrow the automated pipeline that turns issues into tested tasks.
- [SWE-smith](https://mahalsystems.ai/aot/evals/benchmarks/swe-smith/) (2025; coding): Fits teams that need many cheap coding cases from their own repository: point it at a repo, it breaks tests and writes issues. Borrow the bug-injection and keep-only-if-a-test-fails filter.
- [SWR-Bench](https://mahalsystems.ai/aot/evals/benchmarks/swr-bench/) (2025; code-review): Fits pull-request review bots; borrow the structured ground-truth issue list and the coverage-based judge (about 90 percent agreement with humans) as a review rubric.
- [TaxCalcBench](https://mahalsystems.ai/aot/evals/benchmarks/taxcalcbench/) (2025; government, finance, documents): Fits tax-preparation, benefits-eligibility or any form-filling agent whose output is a set of numbered lines. Borrow its line-by-line strict and lenient (plus or minus 5 dollars) scoring, pass^k over repeat runs, and fully synthetic taxpayer cases.
- [TeleMath](https://mahalsystems.ai/aot/evals/benchmarks/telemath/) (2025; telecom, knowledge-qa): Fits radio-planning or network-engineering assistants that must do arithmetic on telecom quantities. Borrow its seed-and-expand pattern (50 expert problems grown to 500 with numeric-only answers that grade without a judge).
- [Terminal-Bench 1.0 (Core v0.1)](https://mahalsystems.ai/aot/evals/benchmarks/terminal-bench-1/) (2025; devops, coding, it-ops): The original terminal task format (task.yaml, Dockerfile, solution.sh, test script) is a clean template for any shell-based agent case; borrow the oracle-solution check.
- [Terminal-Bench 2.0](https://mahalsystems.ai/aot/evals/benchmarks/terminal-bench-2/) (2025; devops, coding, it-ops): The standard number for terminal and devops agents; frontier agents were under 65 percent at release. Borrow Harbor's container-per-task layout and the human-written oracle solution.
- [Tool Decathlon (Toolathlon)](https://mahalsystems.ai/aot/evals/benchmarks/toolathlon/) (2025; tool-calling, workflow-automation, email-calendar, general-assistant, devops): Fits agents connected to many SaaS tools over MCP. Borrow the per-task evaluation scripts that check end state in the real app and the practice of seeding realistic initial states.
- [ToolComp](https://mahalsystems.ai/aot/evals/benchmarks/toolcomp/) (2025; tool-calling, research, finance): Fits research-style assistants that chain several tools to reach one answer; borrow its idea of human-corrected step labels (process supervision) next to a final-answer check, and its enterprise versus chat split.
- [ToolDial](https://mahalsystems.ai/aot/evals/benchmarks/tooldial/) (2025; general-assistant, tool-calling, knowledge-qa): Fits assistants that must ask for missing information before calling an API. The action vocabulary (request, clarify, fail inform) and the API-graph trick for generating dependent calls are worth copying into a case generator.
- [ToolHop](https://mahalsystems.ai/aot/evals/benchmarks/toolhop/) (2025; knowledge-qa, tool-calling, research): Fits research or lookup agents that must chain calls where each step depends on the last. Borrow the query-driven method of writing the tools after the question so every case has a verifiable answer.
- [TutorBench](https://mahalsystems.ai/aot/evals/benchmarks/tutorbench/) (2025; education, knowledge-qa): Fits high-school and AP-level tutoring assistants. Borrow the per-sample rubric pattern, where each prompt ships with its own expert-written checklist that an LLM judge ticks off.
- [UI-CUBE](https://mahalsystems.ai/aot/evals/benchmarks/ui-cube/) (2025; computer-use, workflow-automation, crm): Fits RPA-style enterprise agents; borrow the split into simple interactions versus multi-step workflows and the resolution and widget-variant sweep.
- [UltraHorizon](https://mahalsystems.ai/aot/evals/benchmarks/ultrahorizon/) (2025; planning, general-assistant, research): Fits teams stress-testing agents on very long sessions (200k+ tokens, 400+ calls). Borrow the eight-way error taxonomy (in-context locking versus capability gaps).
- [Vending-Bench](https://mahalsystems.ai/aot/evals/benchmarks/vending-bench/) (2025; planning, finance, retail): Fits long-running operations agents where the question is coherence over weeks of simulated time. Borrow the single hard metric (ending bank balance) and the failure-mode list (forgotten orders, misread schedules, meltdown loops).
- [WASP](https://mahalsystems.ai/aot/evals/benchmarks/wasp/) (2025; safety, security, web-browsing): Fits web agents that act on sites with user-generated content. Borrow the two-level metric, intermediate attack success (judge sees the agent diverted) versus end-to-end success (final state check), which separates gullibility from damage.
- [WebArena-Verified](https://mahalsystems.ai/aot/evals/benchmarks/webarena-verified/) (2025; web-browsing, retail, devops, workflow-automation): Use instead of original WebArena for any browser agent on internal web apps. Borrow the deterministic, type-aware comparators that replaced substring matching and LLM judging.
- [WebChoreArena](https://mahalsystems.ai/aot/evals/benchmarks/webchorearena/) (2025; web-browsing, retail, devops, data-analysis): Fits back-office browser agents that must aggregate data across many pages. Borrow the memory / calculation / long-term-memory task categories.
- [WebGames](https://mahalsystems.ai/aot/evals/benchmarks/webgames/) (2025; web-browsing, games): Fits teams that want fast, hermetic checks of raw browser control. Borrow the password-on-success grading, which needs no judge or state inspection.
- [WebMall](https://mahalsystems.ai/aot/evals/benchmarks/webmall/) (2025; shopping, retail, web-browsing): Fits comparison-shopping agents; borrow the multi-shop offline sandbox built from Common Crawl product pages and the basic / advanced task split.
- [WebWalkerQA](https://mahalsystems.ai/aot/evals/benchmarks/webwalkerqa/) (2025; web-browsing, knowledge-qa): Fits agents that must navigate inside one site (a company portal, a university site, a conference site) rather than search the open web. Borrow the root-site plus question design and the action-count efficiency metric.
- [Who&When](https://mahalsystems.ai/aot/evals/benchmarks/who-and-when/) (2025; multi-agent, general-assistant): Fits teams that want an automated judge to blame the right agent and step after a multi-agent failure. Borrow the agent-plus-step labeling scheme; the reported baselines (about half right on agent, far lower on step) set expectations.
- [WideSearch](https://mahalsystems.ai/aot/evals/benchmarks/widesearch/) (2025; research, web-browsing, data-analysis): Fits agents that build lists or spreadsheets from the web (lead lists, product comparisons, compliance registers). Borrow the table grader: per-column match rules (exact, numeric, date, URL or judge), row and cell F1, and a strict all-cells-correct success rate.
- [xbench-DeepSearch](https://mahalsystems.ai/aot/evals/benchmarks/xbench-deepsearch/) (2025; web-browsing, knowledge-qa, research): Fits Chinese-language search assistants that need a refreshed test set. Borrow the encrypted release with canary strings and the plan to re-issue a new 100-question set periodically.
- [τ²-bench](https://mahalsystems.ai/aot/evals/benchmarks/tau2-bench/) (2025; customer-support, telecom, retail, travel, tool-calling): Fits technical-support agents where the customer must act on their own device or account (dual control), and any policy-bound support agent that writes to a backend. Borrow the compositional task generator (atomic subtasks with init, solution and assertion functions), the assertion-based end-state check, and the error split between reasoning and communication.
- [3CB (Catastrophic Cyber Capabilities Benchmark)](https://mahalsystems.ai/aot/evals/benchmarks/threecb/) (2024; security, it-ops): Fits offensive-security agents and red-team tooling. Borrow the per-task config that bundles sandbox, prompt elicitations and success rule in one file, and the mapping of tasks to ATT&CK techniques.
- [Agent Security Bench (ASB)](https://mahalsystems.ai/aot/evals/benchmarks/agent-security-bench/) (2024; safety, security, tool-calling, memory): Fits teams comparing defenses across attack stages (query, observation, memory, plan). Borrow the attack-by-stage matrix and the utility-versus-security metric.
- [Agent-SafetyBench](https://mahalsystems.ai/aot/evals/benchmarks/agent-safetybench/) (2024; safety, tool-calling, general-assistant): Fits general tool-using assistants that need a broad risk taxonomy. Borrow the 8 risk categories and 10 failure modes, and the idea of a small fine-tuned judge (ShieldAgent) for safe/unsafe labels.
- [AgentBoard](https://mahalsystems.ai/aot/evals/benchmarks/agentboard/) (2024; general-assistant, embodied, games, web-browsing, tool-calling): Fits teams who want partial credit instead of pass/fail for multi-step tasks. Borrow the progress-rate metric (sub-goals reached) and the grounding-accuracy breakdown.
- [AgentClinic](https://mahalsystems.ai/aot/evals/benchmarks/agentclinic/) (2024; healthcare, multi-agent, knowledge-qa): Fits diagnostic or triage assistants that must gather information by asking questions rather than reading a full vignette. Borrow the four-agent setup (patient, doctor, measurement, moderator), the bias injection options, and the structured case JSON.
- [AgentDojo](https://mahalsystems.ai/aot/evals/benchmarks/agentdojo/) (2024; safety, security, tool-calling, email-calendar, finance): Fits assistants that read untrusted content (email, documents, web) and then act with tools. Borrow the pairing of a user task with an injection task, and the utility-under-attack plus targeted attack success rate metrics checked on environment state, not on model text.
- [AgentGym](https://mahalsystems.ai/aot/evals/benchmarks/agentgym/) (2024; general-assistant, embodied, web-browsing, tool-calling, games): Fits teams who want many environments behind one HTTP interface for both training and evaluation of a general agent. Borrow the server-per-environment layout and the AgentEval held-out split.
- [AgentHarm](https://mahalsystems.ai/aot/evals/benchmarks/agentharm/) (2024; safety, tool-calling, general-assistant): Fits any tool-using assistant that must refuse misuse without losing the ability to do the benign version. Borrow the harmful/benign task pairs and the split scoring (refusal judge, semantic judge, tool-call grading functions).
- [Agents4PLC benchmark](https://mahalsystems.ai/aot/evals/benchmarks/agents4plc/) (2024; manufacturing, coding): Fits PLC or industrial-control code-generation agents. Borrow its three-stage gate (compiles, verifiable, passes formal properties) and its task format of requirement plus human-verified formal spec plus reference Structured Text.
- [Aider polyglot benchmark](https://mahalsystems.ai/aot/evals/benchmarks/aider-polyglot/) (2024; coding): Fits chat-style code editing tools that apply diffs to files; borrow the two-attempt protocol (pass rate 1 vs 2) that measures recovery from test feedback.
- [AndroidControl](https://mahalsystems.ai/aot/evals/benchmarks/androidcontrol/) (2024; mobile): Fits teams training mobile action models; borrow the paired high-level and low-level instruction per step and the in-domain / out-of-domain split.
- [AndroidLab](https://mahalsystems.ai/aot/evals/benchmarks/androidlab/) (2024; mobile): Fits mobile agents that run without network or logins. Borrow the four metrics: success, sub-goal success, reversed redundancy and reasonable operation ratio.
- [AndroidWorld](https://mahalsystems.ai/aot/evals/benchmarks/androidworld/) (2024; mobile, general-assistant): Fits mobile app agents; borrow parameterized tasks whose setup, success check and teardown read the device state, giving unlimited variants of 116 templates.
- [API-BLEND](https://mahalsystems.ai/aot/evals/benchmarks/api-blend/) (2024; tool-calling, travel, general-assistant): Fits teams fine-tuning a model for API calling in travel or assistant domains. Borrow the three metrics (API-F1, parameter-F1, LCS-F1 for order) and the method of converting existing dialogue corpora into API-call cases.
- [AppWorld](https://mahalsystems.ai/aot/evals/benchmarks/appworld/) (2024; general-assistant, tool-calling, email-calendar, shopping, planning): Fits assistants that drive several apps through APIs on a user's behalf. Borrow its database-state unit tests that accept any valid path and also check for unwanted side effects.
- [AssistantBench](https://mahalsystems.ai/aot/evals/benchmarks/assistantbench/) (2024; web-browsing, research, general-assistant, shopping): Fits research assistants that browse the live web for a user. Borrow the answer-rate versus accuracy split so that abstaining is counted differently from being wrong.
- [AutoDCWorkflow benchmark](https://mahalsystems.ai/aot/evals/benchmarks/autodcworkflow/) (2024; data-analysis, workflow-automation): Fits data cleaning or data preparation agents that take a messy table and a goal. Borrow its three-way grading: does the clean table answer the purpose, how close are the cell values to the gold table, and how close is the operation sequence to a human workflow.
- [B-MoCA](https://mahalsystems.ai/aot/evals/benchmarks/b-moca/) (2024; mobile): Fits mobile agents that must survive device variation. Borrow the configuration randomizer and the rule-based success detector over system logs, app data and view hierarchy.
- [BALROG](https://mahalsystems.ai/aot/evals/benchmarks/balrog/) (2024; games, planning): Fits teams studying long-horizon decision making with partial credit; borrow the per-game progress metrics that give signal even when a run never finishes.
- [BFCL (Berkeley Function Calling Leaderboard)](https://mahalsystems.ai/aot/evals/benchmarks/bfcl/) (2024; tool-calling, memory, web-browsing): Fits any agent whose core job is choosing and filling tool calls; the AST check for single-turn calls and the per-turn state-plus-trajectory check for multi-turn tasks are the patterns to copy, and the leaderboard gives a current baseline for most hosted models.
- [BigCodeBench](https://mahalsystems.ai/aot/evals/benchmarks/bigcodebench/) (2024; coding, tool-calling): A function-level baseline for library-heavy Python; borrow its high-coverage test writing (about 5.6 tests per task, 99 percent branch coverage).
- [BLADE](https://mahalsystems.ai/aot/evals/benchmarks/blade/) (2024; science, data-analysis): Fits research or analytics agents whose output is an analysis plan, not a single number. Borrow the way expert analysts' decisions (constructs, transforms, model) are collected as ground truth and matched to the agent's choices.
- [BrowserGym / AgentLab](https://mahalsystems.ai/aot/evals/benchmarks/browsergym/) (2024; web-browsing, workflow-automation): A harness, not a task set; fits teams that want to run several web benchmarks with one agent loop. Borrow the gym-style observation and action space and AgentLab's trace viewer.
- [ChemBench](https://mahalsystems.ai/aot/evals/benchmarks/chembench/) (2024; science, knowledge-qa, safety): Fits chemistry assistants and lab-support agents that must answer questions and know when they are unsure. Borrow its topic-level reports, numeric-tolerance parsing of answers, and the comparison against a human chemist panel.
- [CliBench](https://mahalsystems.ai/aot/evals/benchmarks/clibench/) (2024; healthcare, knowledge-qa): Fits coding, order-set or discharge-summary assistants that output sets of standard codes. Borrow the multi-granular scoring that maps predicted codes to ontology ancestors and reports precision, recall and F1 at each level.
- [ClinicalBench (Can LLMs Beat Traditional ML Models in Clinical Prediction?)](https://mahalsystems.ai/aot/evals/benchmarks/clinicalbench/) (2024; healthcare, data-analysis): Fits teams tempted to use an LLM for risk prediction over structured records. Borrow the matched-cohort comparison against 11 traditional ML baselines and the rule that unparseable answers count as wrong.
- [Commit0](https://mahalsystems.ai/aot/evals/benchmarks/commit0/) (2024; coding): Fits greenfield coding agents that build from a spec rather than patch; borrow the "percent of tests passed" partial credit and the lint/type-check gates.
- [CORE-Bench](https://mahalsystems.ai/aot/evals/benchmarks/core-bench/) (2024; research, coding, data-analysis): Fits reproducibility or analysis agents; borrow the question-and-answer report.json grading that avoids judging code directly.
- [CorpFin (Vals AI)](https://mahalsystems.ai/aot/evals/benchmarks/corpfin/) (2024; finance, legal, documents, knowledge-qa): Fits long-document review agents in corporate lending or legal operations. Borrow the question categories (extraction, interpretation, cross-reference, numeric reasoning, market standard) when writing cases over 200-page contracts.
- [CRAB](https://mahalsystems.ai/aot/evals/benchmarks/crab/) (2024; computer-use, mobile, multi-agent): Fits agents that must coordinate across a phone and a computer. Borrow the graph evaluator that gives partial credit for sub-goals reached in any valid order.
- [CRAG (Comprehensive RAG Benchmark)](https://mahalsystems.ai/aot/evals/benchmarks/crag/) (2024; knowledge-qa, research, tool-calling): Fits retrieval-augmented question answering agents that call search or knowledge APIs. Borrow its mock-API sandbox (fixed search results so runs are repeatable) and its scoring that gives +1 for correct, 0 for missing and -1 for hallucinated answers.
- [CRMArena](https://mahalsystems.ai/aot/evals/benchmarks/crmarena/) (2024; crm, customer-support, knowledge-qa, tool-calling): Fits agents that read a real CRM (Salesforce or similar) to answer service, analyst or manager questions. Borrow the persona-based task taxonomy, the 30% unanswerable queries, and the way a synthetic org is generated from a schema with hidden latent variables that drive the ground truth.
- [CRUXEval](https://mahalsystems.ai/aot/evals/benchmarks/cruxeval/) (2024; coding): Fits teams checking whether a model actually reads code before editing it; borrow the execution-prediction task as a cheap probe inside a bigger code-review suite.
- [Cybench](https://mahalsystems.ai/aot/evals/benchmarks/cybench/) (2024; security, coding): Fits offensive-security or CTF-style agents and teams that want a known capability baseline. Borrow the subtask mode that gives partial credit for intermediate steps and the per-task human solve time as a difficulty scale.
- [DA-Code](https://mahalsystems.ai/aot/evals/benchmarks/da-code/) (2024; data-analysis, coding, ml-engineering): Fits data-analysis or notebook agents that write and run code over files. Borrow its three task categories (data wrangling, machine learning, exploratory analysis), the Docker sandbox and the evaluate.py comparison of output files to gold files.
- [DevBench (DevEval)](https://mahalsystems.ai/aot/evals/benchmarks/devbench/) (2024; coding): Fits agents covering more than coding, such as design docs and test writing; borrow its lifecycle stages as a taxonomy for an agent's golden set.
- [DiscoveryBench](https://mahalsystems.ai/aot/evals/benchmarks/discoverybench/) (2024; science, data-analysis, research): Fits research agents that must turn a dataset and a goal into a written finding. Borrow the faceted judge: split gold and predicted hypotheses into context, variables and relationship and score each part separately.
- [DocBench](https://mahalsystems.ai/aot/evals/benchmarks/docbench/) (2024; documents, knowledge-qa, finance, legal): Fits document question-answering assistants over contracts, filings or papers. Borrow the four question types (including unanswerable) over 229 real documents in five domains.
- [DSBench](https://mahalsystems.ai/aot/evals/benchmarks/dsbench/) (2024; data-analysis, ml-engineering, coding): Fits data science agents that must read long briefs and large or multi-table files, and agents that build models end to end. Borrow the relative performance gap metric that scores a model against the human leaderboard on the same Kaggle task.
- [DSEval](https://mahalsystems.ai/aot/evals/benchmarks/dseval/) (2024; data-analysis, coding): Fits notebook-style assistants where each question builds on the last. Borrow the problem set format (ordered problems sharing one session) and the bootstrapped annotation method for expanding cases.
- [EHRAgent (EHRSQL-derived agent evaluation)](https://mahalsystems.ai/aot/evals/benchmarks/ehragent/) (2024; healthcare, sql, data-analysis, tool-calling): Fits data-retrieval agents that let clinicians ask questions of structured EHR tables in plain language. Borrow the code-as-plan action format, the interactive debugging loop, and the question complexity levels (number of tables joined).
- [EHRNoteQA](https://mahalsystems.ai/aot/evals/benchmarks/ehrnoteqa/) (2024; healthcare, documents, knowledge-qa): Fits chart-summarization or note question-answering assistants that read long discharge summaries. Borrow the clinician-reviewed question generation over multiple notes and the reported correlation with clinician judgments as a way to validate a proxy benchmark.
- [FinBen](https://mahalsystems.ai/aot/evals/benchmarks/finben/) (2024; finance, knowledge-qa, data-analysis): Fits teams that need a broad baseline for a finance language model behind an agent rather than the agent itself. Borrow its grouping of 36 datasets into 24 task types to check which skills your model is missing.
- [FRAMES](https://mahalsystems.ai/aot/evals/benchmarks/frames/) (2024; knowledge-qa, research): Fits research or multi-hop retrieval agents that must chain facts across documents. Borrow its per-question reasoning-type labels and the inspect_evals Socrates scoring, which discounts accuracy by extra retrieval hops and penalizes requests for documents that do not exist.
- [FunctionChat-Bench](https://mahalsystems.ai/aot/evals/benchmarks/functionchat-bench/) (2024; general-assistant, tool-calling): Fits chat assistants where every turn must be one of call, answer, ask, or refuse. The four-way output taxonomy and the rubric-based pass/fail judge are a compact grading design to reuse.
- [GDM Dangerous Capabilities: Capture the Flag](https://mahalsystems.ai/aot/evals/benchmarks/gdm-dangerous-capabilities-ctf/) (2024; security, it-ops): Fits offensive-security agents and teams that need a small in-house CTF set that is not on the public internet. Borrow the attacker-plus-victim compose layout and the at-least-one-of-N epochs reducer.
- [GDM Dangerous Capabilities: Self-proliferation](https://mahalsystems.ai/aot/evals/benchmarks/gdm-dangerous-capabilities-self-proliferation/) (2024; safety, security, web-browsing, it-ops): Fits teams giving an agent real web, cloud and shell access who need to know how far it can get on its own. Borrow the milestone mode that scores partial progress and the human-approval gate on every tool call.
- [GTA (General Tool Agents)](https://mahalsystems.ai/aot/evals/benchmarks/gta/) (2024; tool-calling, general-assistant, data-analysis): Fits assistants whose users send screenshots, photos or files and expect the right tools to be picked without being told. Borrow the human-written implicit-tool-use queries and the split between step-level tool accuracy and final-answer accuracy.
- [HammerBench](https://mahalsystems.ai/aot/evals/benchmarks/hammerbench/) (2024; mobile, general-assistant, tool-calling): Fits on-device or app assistants doing slot filling. Borrow the per-turn snapshot metrics (parameter hallucination rate, parameter missing rate, progress rate) and the four perturbation types (imperfect, Q&A trajectory, intent shift, external reference).
- [HR-MultiWOZ](https://mahalsystems.ai/aot/evals/benchmarks/hr-multiwoz/) (2024; hr, customer-support): Fits HR help-desk chatbots that need conversation test cases. Borrow the LLM-generated dialogue pipeline built to avoid real HR conversations.
- [IaC-Eval](https://mahalsystems.ai/aot/evals/benchmarks/iac-eval/) (2024; devops, it-ops): Fits infrastructure-as-code agents; borrow intent specs checked with OPA against the Terraform plan instead of string-matching the HCL.
- [InfiAgent-DABench](https://mahalsystems.ai/aot/evals/benchmarks/infiagent-dabench/) (2024; data-analysis, coding): Fits a data analysis assistant that answers questions over uploaded CSV files with code. Borrow the trick of rewriting open-ended questions into closed-form answers so grading is an exact match.
- [InjecAgent](https://mahalsystems.ai/aot/evals/benchmarks/injecagent/) (2024; safety, security, tool-calling): Fits a first, cheap check of any tool-calling agent for indirect prompt injection. Borrow the two attacker goals (direct harm, data theft) and the user-case times attacker-case crossing that gives many cases from few templates.
- [LAB-Bench](https://mahalsystems.ai/aot/evals/benchmarks/lab-bench/) (2024; science, knowledge-qa, research): Fits research-assistant agents for biology that answer questions over papers, databases and sequences. Borrow the eight-category taxonomy (LitQA2, DbQA, SuppQA, FigQA, TableQA, ProtocolQA, SeqQA, Cloning) and the accuracy-plus-coverage reporting that rewards abstaining.
- [LegalAgentBench](https://mahalsystems.ai/aot/evals/benchmarks/legalagentbench/) (2024; legal, tool-calling, knowledge-qa, documents): Fits legal research or due-diligence agents that query several databases and write up the result. Borrow its progress-rate grading (keyword checks on annotated intermediate steps) and its tool-over-corpus sandbox.
- [LiveCodeBench](https://mahalsystems.ai/aot/evals/benchmarks/livecodebench/) (2024; coding): A model-level coding baseline rather than an agent test; borrow the date-windowed release scheme that lets you report only problems newer than a model's cutoff.
- [LoCoMo](https://mahalsystems.ai/aot/evals/benchmarks/locomo/) (2024; memory, knowledge-qa): Fits assistants that must remember facts a user told them weeks ago across sessions. Borrow the persona-plus-event-graph recipe for generating long, consistent fake conversations without real user data.
- [Long Code Arena](https://mahalsystems.ai/aot/evals/benchmarks/long-code-arena/) (2024; coding, devops): Six project-wide tasks in one suite; the CI build repair task, graded by re-running the real GitHub Actions workflow, is a usable pattern for devops agents.
- [LongBench v2](https://mahalsystems.ai/aot/evals/benchmarks/longbench-v2/) (2024; documents, knowledge-qa, coding, research): Fits any agent that must read long inputs (8k to 2M words) before answering, such as contract review or codebase question answering. Borrow its six-category task split and its human-written, time-limited question style, which keeps questions hard to shortcut.
- [LongMemEval](https://mahalsystems.ai/aot/evals/benchmarks/longmemeval/) (2024; memory, knowledge-qa, general-assistant): Fits chat assistants with long-term memory across sessions; the five ability categories (extraction, multi-session reasoning, temporal, updates, abstention) are a ready taxonomy for a memory test set. Borrow the needle-in-history construction and the 30 abstention cases.
- [m&m's](https://mahalsystems.ai/aot/evals/benchmarks/mnms/) (2024; tool-calling, planning, documents, general-assistant): Fits agents that plan a short pipeline over heterogeneous tools (OCR, captioning, weather, search) before executing. Borrow its plan-level metrics (tool-F1, argname-F1, pass rate) and its finding that step-by-step planning with feedback beats one-shot plans.
- [Make Me Pay](https://mahalsystems.ai/aot/evals/benchmarks/make-me-pay/) (2024; multi-agent, safety): Fits payment or negotiation agents that talk to other agents or users and must not be manipulated (or manipulate). Borrow the two-model setup where the outcome (money moved) is checked programmatically rather than judged.
- [MedCalc-Bench](https://mahalsystems.ai/aot/evals/benchmarks/medcalc-bench/) (2024; healthcare, knowledge-qa, data-analysis): Fits clinical decision-support assistants that must compute scores or doses from notes. Borrow the tolerance-band grading (plus or minus 5 percent for decimal outputs), the relevant-entities field for partial-credit diagnosis, and the 55 calculator taxonomy.
- [MedChain](https://mahalsystems.ai/aot/evals/benchmarks/medchain/) (2024; healthcare, multi-agent, knowledge-qa): Fits end-to-end clinical workflow agents where an early mistake (wrong specialty, missed question) should hurt later stages. Borrow the five-stage sequential case format and the per-stage metrics (accuracy and IoU for referral and history, LLM scoring for diagnosis).
- [Mind2Web-SC](https://mahalsystems.ai/aot/evals/benchmarks/mind2web-sc/) (2024; safety, web-browsing): Fits teams adding a policy guardrail in front of a web or booking agent. Borrow the rule-plus-profile case format and the split accuracy (right decision, right violation named, right action kept).
- [MLE-bench](https://mahalsystems.ai/aot/evals/benchmarks/mle-bench/) (2024; ml-engineering, data-analysis): The standard test for ML-engineering agents; borrow medal-threshold grading against a human leaderboard and the 22-competition Lite split for cheaper runs.
- [MTU-Bench](https://mahalsystems.ai/aot/evals/benchmarks/mtu-bench/) (2024; travel, general-assistant, tool-calling): Fits booking and lookup agents built on slot-filling dialogues. Borrow its 2x2 taxonomy (single or multi turn by single or multi tool) and its metric set (tool selection, parameter accuracy, turn success, task progress) that needs no LLM judge.
- [MultiHop-RAG](https://mahalsystems.ai/aot/evals/benchmarks/multihop-rag/) (2024; knowledge-qa, research): Fits retrieval-augmented agents over a corpus where one question spans several documents. Borrow its split of retrieval scoring (did the right evidence come back) from answer scoring, and its four query types including null queries with no answer in the corpus.
- [NESTFUL](https://mahalsystems.ai/aot/evals/benchmarks/nestful/) (2024; tool-calling): Fits agents that must pass results between tools in one plan (data pipelines, calculators, code helpers); borrow its label-and-reference format for nested calls and its metric ladder (API-name F1, parameter F1, partial and full sequence match, executed win rate).
- [NoisyToolBench](https://mahalsystems.ai/aot/evals/benchmarks/noisytoolbench/) (2024; tool-calling, general-assistant): Fits any agent that gets vague requests and should ask rather than guess. Borrow the case format (imperfect query, expected clarifying question, scripted answer, expected call) and the ToolEvaluator that scores both accuracy and redundant questions.
- [OfficeBench](https://mahalsystems.ai/aot/evals/benchmarks/officebench/) (2024; documents, spreadsheets, email-calendar, general-assistant): Fits office assistants that move content between documents, spreadsheets, email and calendars. Borrow the three-tier design (one, two, three apps) and the mix of exact, fuzzy and execution-based checks on output files.
- [OmniACT](https://mahalsystems.ai/aot/evals/benchmarks/omniact/) (2024; computer-use, web-browsing): Fits teams training grounding-plus-planning in one shot. Borrow the sequence score and action score (click-in-box, key set, typed-text BLEU).
- [OmniDocBench](https://mahalsystems.ai/aot/evals/benchmarks/omnidocbench/) (2024; documents): Fits the parsing step of any document pipeline, where PDFs become text for an agent to reason over. Borrow its per-element metrics (edit distance for text, TEDS for tables, CDM for formulas) and its page attribute labels for slicing results by layout and language.
- [OSWorld](https://mahalsystems.ai/aot/evals/benchmarks/osworld/) (2024; computer-use, documents, spreadsheets, it-ops): The reference for desktop agents; borrow the per-task setup config plus execution-based evaluator script pattern and the VM snapshot reset.
- [QRData](https://mahalsystems.ai/aot/evals/benchmarks/qrdata/) (2024; data-analysis, science, education): Fits analysis assistants that must apply statistics or causal methods correctly, not just run code. Borrow the pairing of each question with a data sheet and a companion text-only set to tell reasoning failures from data-handling failures.
- [RAGBench](https://mahalsystems.ai/aot/evals/benchmarks/ragbench/) (2024; knowledge-qa, finance, healthcare, legal, customer-support): Fits teams building the judge for a retrieval-augmented agent rather than the agent itself. Borrow its TRACe labels (relevance, utilization, adherence, completeness) at sentence level as a rubric for your own answer grader, and use its 12 sub-datasets to test that grader.
- [RE-Bench](https://mahalsystems.ai/aot/evals/benchmarks/re-bench/) (2024; ml-engineering, research): Fits ML research-engineering agents; its value is the 71 human-expert 8-hour attempts that give a time-budgeted human baseline to compare against.
- [RoTBench](https://mahalsystems.ai/aot/evals/benchmarks/rotbench/) (2024; tool-calling, general-assistant): Fits teams whose tool schemas change or are poorly documented. The pattern to copy is taking a clean case set and producing slight, medium, heavy and union noise variants to measure robustness.
- [SciCode](https://mahalsystems.ai/aot/evals/benchmarks/scicode/) (2024; science, coding): Fits scientific-coding assistants; borrow the main-problem versus subproblem scoring so partial progress is visible.
- [ScienceAgentBench](https://mahalsystems.ai/aot/evals/benchmarks/scienceagentbench/) (2024; science, data-analysis, coding, research): Fits data-analysis or scientific coding agents that take a dataset and a goal and must hand back a runnable program. Borrow its per-task evaluation scripts that execute the generated code and compare outputs, plus the best-of-three-runs scoring.
- [SciKnowEval](https://mahalsystems.ai/aot/evals/benchmarks/sciknoweval/) (2024; science, knowledge-qa, safety): Fits science question-answering assistants in biology, chemistry, physics or materials. Borrow the five-level taxonomy (memory, comprehension, reasoning, discernment, application) as a way to bucket an agent's own test cases by depth.
- [Seal-Tools](https://mahalsystems.ai/aot/evals/benchmarks/seal-tools/) (2024; tool-calling): Fits teams who need a large, fully made-up tool catalogue for fine-tuning or retrieval tests; borrow the self-instruct pipeline (field, tool, instance) and its format accuracy plus tool and parameter precision/recall/F1 scoring.
- [ShortcutsBench](https://mahalsystems.ai/aot/evals/benchmarks/shortcutsbench/) (2024; workflow-automation, tool-calling, mobile, planning): Fits workflow-automation agents that compose many app or SaaS APIs into one long sequence. Borrow its four-way grading (API choice, parameter fill, reuse of prior outputs, asking for input) and its difficulty tiers by sequence length.
- [SimpleQA](https://mahalsystems.ai/aot/evals/benchmarks/simpleqa/) (2024; knowledge-qa): Fits assistants whose risk is confidently stating wrong facts. Borrow its three-way grade (correct, incorrect, not attempted) and its F-score that rewards abstaining over guessing; this is a good pattern for any agent that should say "I don't know".
- [Spider 2.0](https://mahalsystems.ai/aot/evals/benchmarks/spider2/) (2024; sql, data-analysis, coding): Fits analytics or warehouse agents that must answer questions over big real schemas in cloud SQL dialects. Borrow its three sub-variants: Spider2-Lite (547 tasks, BigQuery 214, Snowflake 198, SQLite 135), Spider2-Snow (547 tasks on Snowflake only) and Spider2-DBT (68 repository-level dbt tasks on DuckDB), and its execution-result grading.
- [Spider2-V](https://mahalsystems.ai/aot/evals/benchmarks/spider2-v/) (2024; computer-use, data-analysis, devops, sql): Fits agents that operate data tools through a GUI or mixed GUI plus code, such as a pipeline operator or warehouse admin assistant. Borrow its per-task evaluation functions and the VM built on OSWorld with human-validation scripts.
- [SpreadsheetBench](https://mahalsystems.ai/aot/evals/benchmarks/spreadsheetbench/) (2024; spreadsheets, data-analysis, coding): Fits spreadsheet or Excel assistants that edit workbooks from a user request. Borrow the online-judge pattern: about three input workbooks per instruction, and the output cells must match on every one.
- [ST-WebAgentBench](https://mahalsystems.ai/aot/evals/benchmarks/st-webagentbench/) (2024; safety, web-browsing, crm, devops): Fits browser agents that work inside company systems under rules, such as CRM or admin-panel automation. Borrow the Completion Under Policy metric (a task only counts if no policy was broken), the six policy dimensions, and the per-policy evaluator pattern.
- [StableToolBench](https://mahalsystems.ai/aot/evals/benchmarks/stabletoolbench/) (2024; tool-calling, general-assistant): Fits agents that call many third-party web APIs and need a replayable test bed; the cache plus LLM-simulated-API server is the pattern to copy when the real APIs are flaky, and the solvable-only filtering shows how to prune unanswerable cases.
- [SWE-bench Lite](https://mahalsystems.ai/aot/evals/benchmarks/swe-bench-lite/) (2024; coding): A cheap smoke test for a coding agent: single-file edits, at most three hunks, no images or links in the issue. Borrow its filter rules when you need a small, clean subset of a bigger set.
- [SWE-bench Multimodal](https://mahalsystems.ai/aot/evals/benchmarks/swe-bench-multimodal/) (2024; coding): Fits agents that work on front-end or visualization code where the bug report is a picture. Borrow the way image inputs are attached to an otherwise text-only task.
- [SWE-bench Verified](https://mahalsystems.ai/aot/evals/benchmarks/swe-bench-verified/) (2024; coding): The default number to compare against for issue-to-patch coding agents in Python. Borrow its human screening step, where engineers rejected tasks whose issue text or tests were unfair.
- [SWE-bench-java](https://mahalsystems.ai/aot/evals/benchmarks/swe-bench-java/) (2024; coding): An early Java-only issue set; useful if your agent targets JVM codebases and you want a tiny hand-verified pilot. Borrow the Docker-per-instance build setup for Maven projects.
- [SWE-Gym](https://mahalsystems.ai/aot/evals/benchmarks/swe-gym/) (2024; coding): A training environment rather than a test set; fits teams fine-tuning their own coding agent or verifier. Borrow the prebuilt per-instance Docker images and the Lite split of 234 tasks.
- [TableBench](https://mahalsystems.ai/aot/evals/benchmarks/tablebench/) (2024; data-analysis, knowledge-qa): Fits assistants that answer questions over a single table in a chat. Borrow the 18-category taxonomy under four headings and the per-category metric choice (exact match, exact match with a 10 percent margin, ROUGE-L, pass at 1 for charts).
- [TheAgentCompany](https://mahalsystems.ai/aot/evals/benchmarks/theagentcompany/) (2024; general-assistant, coding, hr, documents, workflow-automation): Fits digital-worker agents that touch several internal tools and must ask coworkers for details. Borrow the self-hosted company sandbox, the LLM-played colleagues and the checkpoint-plus-result scoring.
- [ToolEyes](https://mahalsystems.ai/aot/evals/benchmarks/tooleyes/) (2024; general-assistant, tool-calling, finance, research): Fits general assistants with a broad tool library. Borrow the five-dimension scoring (format, intent, planning, tool selection, answer) that mixes rule checks with a judge instead of grading only the final answer.
- [ToolSandbox](https://mahalsystems.ai/aot/evals/benchmarks/toolsandbox/) (2024; tool-calling, general-assistant, mobile): Fits assistant agents whose tools have hidden state dependencies (one call only works after another). Borrow the milestone and minefield grading: required world-state checkpoints in a partial order, forbidden events that zero the score, and similarity scoring of each checkpoint.
- [TravelPlanner](https://mahalsystems.ai/aot/evals/benchmarks/travelplanner/) (2024; travel, planning, tool-calling): Fits planning assistants that must juggle many constraints. Borrow the split between commonsense and hard constraint pass rates plus a final all-constraints pass rate.
- [UltraTool](https://mahalsystems.ai/aot/evals/benchmarks/ultratool/) (2024; tool-calling, planning, general-assistant): Fits agents that plan multi-step work over a large tool catalog, and teams that want to grade planning separately from execution. Borrow its planning/creation/usage split, its six-dimension GPT-4 plan rubric, and its key-value accuracy for tool selection and arguments.
- [USACO](https://mahalsystems.ai/aot/evals/benchmarks/usaco/) (2024; coding): A cost-tracked algorithmic baseline available on HAL; useful as a sanity check for reasoning-heavy coding agents, not for product engineering.
- [VisualWebArena](https://mahalsystems.ai/aot/evals/benchmarks/visualwebarena/) (2024; web-browsing, retail, shopping): Fits agents that must read screenshots or product images while browsing (classifieds, marketplaces). Borrow the image-conditioned task templates and the execution-based checks.
- [WebCanvas (Mind2Web-Live)](https://mahalsystems.ai/aot/evals/benchmarks/webcanvas/) (2024; web-browsing, shopping, travel): Fits any live-site browser agent where the final page cannot be checked directly. Borrow key-node grading: credit for reaching required URLs or elements mid-task.
- [WebLINX](https://mahalsystems.ai/aot/evals/benchmarks/weblinx/) (2024; web-browsing, general-assistant): Fits chat-driven browser copilots where a human gives instructions turn by turn. Borrow the demonstration format of paired user utterances and browser actions.
- [WebVoyager](https://mahalsystems.ai/aot/evals/benchmarks/webvoyager/) (2024; web-browsing, shopping, travel, knowledge-qa): Fits consumer-facing browser agents on well-known sites. Borrow the screenshot-based auto-judge that reads the last k screenshots.
- [Windows Agent Arena](https://mahalsystems.ai/aot/evals/benchmarks/windows-agent-arena/) (2024; computer-use, documents, it-ops): Fits agents that must run on Windows desktops. Borrow the Azure-parallel runner that finishes the benchmark in about 20 minutes.
- [WorfBench](https://mahalsystems.ai/aot/evals/benchmarks/worfbench/) (2024; planning, tool-calling, embodied, workflow-automation): Fits agents that produce a plan before executing, such as workflow builders. Borrow the graph matching evaluation (F1 on node chain and on the workflow graph) that rewards correct parallel structure, not just step order.
- [WorkArena](https://mahalsystems.ai/aot/evals/benchmarks/workarena/) (2024; it-ops, workflow-automation, web-browsing, computer-use): Fits browser agents working inside ServiceNow or similar ITSM/HR portals. Borrow the task templates (33 atomic tasks instantiated into thousands of variants) and the per-task validate() reward on a live instance.
- [WorkArena++](https://mahalsystems.ai/aot/evals/benchmarks/workarena-plus-plus/) (2024; it-ops, workflow-automation, web-browsing, planning): Fits browser agents doing compositional work in ITSM and HR portals. Borrow its way of building hard tasks by composing atomic ones and its ground-truth trace generator for training data.
- [WorkBench](https://mahalsystems.ai/aot/evals/benchmarks/workbench/) (2024; email-calendar, crm, data-analysis, tool-calling, workflow-automation): Fits assistants that act on email, calendar and CRM data through tools. Borrow the outcome-centric grading that compares final database state regardless of action order and reports side effects separately.
- [τ-bench](https://mahalsystems.ai/aot/evals/benchmarks/tau-bench/) (2024; customer-support, retail, travel, tool-calling): Fits support or booking agents that must follow a written policy while talking to a customer and writing to a backend. Borrow the end-state database diff as the pass/fail check, the pass^k reliability metric, and the simulated user prompted with a hidden user instruction.
- [AgentBench](https://mahalsystems.ai/aot/evals/benchmarks/agentbench/) (2023; general-assistant, it-ops, sql, shopping, games): Fits teams who want one harness covering shell, database and web-style tasks to sanity-check a general agent. Borrow the containerized per-environment servers; the 2025 function-calling version keeps it usable with current tool-call APIs.
- [AgentVerse](https://mahalsystems.ai/aot/evals/benchmarks/agentverse/) (2023; multi-agent, coding, general-assistant): Fits teams prototyping multi-agent orchestration rather than teams looking for a test set; the repo is a framework with a HumanEval runner, useful as a reference for recruit-plan-act-review loops.
- [API-Bank](https://mahalsystems.ai/aot/evals/benchmarks/api-bank/) (2023; tool-calling, general-assistant): Fits assistants that must pick one API out of many and fill its arguments from a chat history. Borrow its three-level split (call, retrieve then call, plan then retrieve then call) and its dialogue-plus-expected-call case format.
- [BIRD](https://mahalsystems.ai/aot/evals/benchmarks/bird/) (2023; sql, data-analysis, knowledge-qa): Fits single-shot text-to-SQL agents that answer questions over one database with a known schema. Borrow its execution-accuracy grading, the evidence hint pattern and the human baseline (92.96 execution accuracy) as a reference point.
- [ClassEval](https://mahalsystems.ai/aot/evals/benchmarks/classeval/) (2023; coding): Fits code generators that emit whole modules; borrow class-level versus method-level pass@1 reporting to show where multi-part outputs break.
- [DocILE](https://mahalsystems.ai/aot/evals/benchmarks/docile/) (2023; documents, finance, workflow-automation): Fits invoice and purchase-order extraction agents. Borrow its 55-class field taxonomy, its line-item grouping task, and its scorer that matches predicted fields to ground truth by bounding-box overlap and reports precision, recall and F1 per field.
- [DocMath-Eval](https://mahalsystems.ai/aot/evals/benchmarks/docmath-eval/) (2023; finance, documents, data-analysis): Fits finance or analyst agents that must compute figures from 10-K and 10-Q filings. Borrow its four-way split (simple vs complex reasoning, short vs long document) and its numeric answer checker that accepts chain-of-thought or program-of-thought output.
- [FinanceBench](https://mahalsystems.ai/aot/evals/benchmarks/financebench/) (2023; finance, documents, knowledge-qa): Fits retrieval-augmented finance assistants that answer questions over company filings. Borrow its case format (question, answer, evidence string, justification, question type, reasoning type) as a template for a golden set over your own document corpus.
- [GAIA](https://mahalsystems.ai/aot/evals/benchmarks/gaia/) (2023; general-assistant, research, web-browsing, knowledge-qa): Fits research-style assistants that must return one checkable answer. Borrow the short-answer exact-match format and the three difficulty levels.
- [Gorilla APIBench](https://mahalsystems.ai/aot/evals/benchmarks/gorilla-apibench/) (2023; tool-calling, ml-engineering, coding): Fits code-writing agents that must call library APIs from documentation; borrow its AST sub-tree match (compare only the arguments that matter) and its separate count of hallucinated versus wrongly used APIs.
- [HumanEvalFix (HumanEvalPack)](https://mahalsystems.ai/aot/evals/benchmarks/humanevalfix/) (2023; coding): A tiny bug-fix template: 164 HumanEval problems times six languages with planted bugs; useful only as a quick multi-language smoke test for a repair agent.
- [InterCode CTF](https://mahalsystems.ai/aot/evals/benchmarks/intercode-ctf/) (2023; security, coding): Fits teams that want a quick, cheap smoke test for a shell-using agent. Borrow the environment-as-RL-loop design where code is the action and execution output is the observation.
- [LawBench](https://mahalsystems.ai/aot/evals/benchmarks/lawbench/) (2023; legal, knowledge-qa, documents): Fits Chinese-law question-answering or document-processing assistants. Borrow its three-level split (memorize, understand, apply) and its abstention-rate metric for answers that cannot be parsed.
- [LegalBench](https://mahalsystems.ai/aot/evals/benchmarks/legalbench/) (2023; legal, knowledge-qa, documents): Fits contract-review, compliance and legal-research assistants that must classify clauses or apply a rule to facts. Borrow its six-way reasoning taxonomy (issue, rule, conclusion, application, interpretation, rhetoric) and its short hand-written task cards.
- [MetaTool](https://mahalsystems.ai/aot/evals/benchmarks/metatool/) (2023; general-assistant, tool-calling): Fits any agent with a large tool catalog where the first failure is picking the wrong tool. Its four selection subtasks (similar, scenario-specific, reliability, multi-tool) are a ready taxonomy for tool-routing test cases.
- [Mind2Web](https://mahalsystems.ai/aot/evals/benchmarks/mind2web/) (2023; web-browsing, shopping, travel, general-assistant): Fits teams training or testing step-level action prediction on real site layouts. Borrow the cross-task / cross-website / cross-domain split design.
- [MINT](https://mahalsystems.ai/aot/evals/benchmarks/mint/) (2023; coding, general-assistant, tool-calling, embodied): Fits agents that iterate with a user on code or quantitative answers. The useful part is the GPT-4 feedback simulator and the turn-budget analysis (k = 1 to 5 turns), which shows how much a feedback loop is worth.
- [RepoBench](https://mahalsystems.ai/aot/evals/benchmarks/repobench/) (2023; coding): Fits autocomplete or inline-suggestion products, not agents; borrow exact match plus edit similarity plus CodeBLEU as cheap text-level grading for completion cases.
- [RestBench](https://mahalsystems.ai/aot/evals/benchmarks/restbench/) (2023; tool-calling, general-assistant, knowledge-qa): Fits agents that call real REST services from OpenAPI docs. Borrow the small hand-annotated gold-path format and the correct-path-rate metric alongside success.
- [SheetCopilot](https://mahalsystems.ai/aot/evals/benchmarks/sheetcopilot/) (2023; spreadsheets, workflow-automation): Fits agents that drive a spreadsheet application through a fixed action set rather than writing code. Borrow the per-task checking file that names which sheet properties to compare against one or more reference workbooks.
- [SWE-bench](https://mahalsystems.ai/aot/evals/benchmarks/swe-bench/) (2023; coding): Fits any agent that fixes bugs or implements changes in an existing codebase from a written issue. Borrow the FAIL_TO_PASS / PASS_TO_PASS test split and the per-task Docker image pattern.
- [Sycophancy Eval](https://mahalsystems.ai/aot/evals/benchmarks/sycophancy-eval/) (2023; safety, knowledge-qa, general-assistant): Fits any agent that answers users who push back, such as support, tutoring or advisory agents. Borrow the "are you sure?" follow-up turn and the three metrics (confidence, apologize rate, truthfulness) as a cheap regression check for caving to pressure.
- [T-Eval](https://mahalsystems.ai/aot/evals/benchmarks/t-eval/) (2023; tool-calling, planning): Fits teams that want to diagnose which step of tool use their agent gets wrong (format, tool choice, plan, reasoning, argument extraction, response review) rather than only the end result. Borrow its decomposition and its per-step graders.
- [TaskBench](https://mahalsystems.ai/aot/evals/benchmarks/taskbench/) (2023; tool-calling, planning, workflow-automation): Fits workflow-automation agents that must plan a tool chain before executing. Borrow the node-F1 and edge-F1 scoring of predicted tool graphs and the back-instruct data generator.
- [TeleQnA](https://mahalsystems.ai/aot/evals/benchmarks/teleqna/) (2023; telecom, knowledge-qa): Fits network-operations or telecom customer-support assistants as a knowledge check before agent testing. Borrow its five-category split (lexicon, research overview, research publications, standards overview, standards specifications).
- [ToolAlpaca](https://mahalsystems.ai/aot/evals/benchmarks/toolalpaca/) (2023; tool-calling, general-assistant): Fits teams that need to test an agent against APIs that do not exist yet or cannot be hit in test; borrow its LLM-simulated API server and its procedure/response/overall grading split. Mostly a training corpus, not a benchmark.
- [ToolBench (ToolLLM)](https://mahalsystems.ai/aot/evals/benchmarks/toolbench/) (2023; tool-calling, general-assistant): Fits agents that must choose from a very large tool catalogue and chain real web APIs; borrow its single-tool / same-category / cross-collection split (G1, G2, G3) for building a tool coverage taxonomy.
- [ToolEmu](https://mahalsystems.ai/aot/evals/benchmarks/toolemu/) (2023; safety, tool-calling, general-assistant, finance): Fits teams that need to test dangerous actions before any real tool exists. Borrow the LLM-emulated tool sandbox and the paired safety and helpfulness scores on a 0-3 scale.
- [ToolQA](https://mahalsystems.ai/aot/evals/benchmarks/toolqa/) (2023; knowledge-qa, tool-calling, data-analysis, sql): Fits question-answering agents over a company's own tables or document store. Borrow its template-plus-sampled-values way of generating questions with programmatically computed answers, and its check that the question cannot be answered without the tool.
- [ToolTalk](https://mahalsystems.ai/aot/evals/benchmarks/tooltalk/) (2023; general-assistant, email-calendar, tool-calling): Fits personal-assistant agents that act on calendars, email and messages in a conversation. Borrow the simulated tool implementations with mock databases and the strict success rule (perfect recall of required calls, zero wrong actions).
- [WebArena](https://mahalsystems.ai/aot/evals/benchmarks/webarena/) (2023; web-browsing, retail, devops, workflow-automation): Fits browser agents that act inside internal web apps (shop admin, issue tracker, forum). Borrow the self-hosted site sandbox and the per-task evaluator mix of exact answer, target URL and page-content program checks.
- [DS-1000](https://mahalsystems.ai/aot/evals/benchmarks/ds-1000/) (2022; data-analysis, coding): Fits a code assistant that writes pandas or numpy snippets for analysts. Borrow the two-part check (run on fresh inputs against a reference solution, plus a token check on the code text) and the practice of perturbing scraped problems to limit memorization.
- [EHRSQL](https://mahalsystems.ai/aot/evals/benchmarks/ehrsql/) (2022; healthcare, sql): Fits text-to-SQL or data-retrieval agents over hospital databases. Borrow the questions collected from 222 real hospital staff, the time-expression templates, the unanswerable question split, and the execution-based F1 that penalizes confident wrong answers.
- [ScienceWorld](https://mahalsystems.ai/aot/evals/benchmarks/scienceworld/) (2022; science, education, embodied, planning): Fits teams who want a procedure-following environment with partial-progress scoring; the 30 task types have about 7,000 parametric variations, so it resists memorization better than ALFWorld.
- [WDC Products](https://mahalsystems.ai/aot/evals/benchmarks/wdc-products/) (2022; data-analysis, retail, shopping): Fits product catalog or deduplication agents that must say whether two records are the same item. Borrow its difficulty dials (share of corner cases, share of unseen entities, training set size) across 27 variants built from 11,715 offers of 2,162 products.
- [WebShop](https://mahalsystems.ai/aot/evals/benchmarks/webshop/) (2022; shopping, retail, web-browsing): Fits shopping assistants; borrow the self-hosted product catalog sandbox and the attribute-and-price reward computed from the bought item.
- [APPS](https://mahalsystems.ai/aot/evals/benchmarks/apps/) (2021; coding): Still used as a large, cheap code-generation baseline; its test-case checker is reused by LiveCodeBench, so it is a known-good grading script.
- [FinQA](https://mahalsystems.ai/aot/evals/benchmarks/finqa/) (2021; finance, documents, data-analysis): Fits finance agents that compute figures from reports and must show their work. Borrow its two-level grading (execution accuracy on the final number plus program accuracy on the reasoning steps) when you want to check reasoning, not just the answer.
- [TAT-QA](https://mahalsystems.ai/aot/evals/benchmarks/tat-qa/) (2021; finance, documents, data-analysis): Fits agents that read financial statements where the answer needs both a table and the surrounding text. Borrow its answer-type tagging (span, multi-span, count, arithmetic) and its exact-match and F1 scorer with scale handling.
- [ALFWorld](https://mahalsystems.ai/aot/evals/benchmarks/alfworld/) (2020; embodied, planning): Fits teams who need a cheap, deterministic multi-step environment for smoke tests of a planning loop; it is also the base of the household tasks in AgentBench, AgentBoard and AgentGym.
- [BugsInPy](https://mahalsystems.ai/aot/evals/benchmarks/bugsinpy/) (2020; coding): A Python counterpart to Defects4J for repair agents; useful as a small, reproducible pilot set before building your own.
- [DocVQA](https://mahalsystems.ai/aot/evals/benchmarks/docvqa/) (2020; documents): Fits agents that read scanned or photographed documents (forms, letters, reports) and pull out a short answer. Borrow its question-over-image case format and its ANLS scoring, which tolerates small OCR spelling errors instead of demanding an exact string.
- [MedQA (USMLE)](https://mahalsystems.ai/aot/evals/benchmarks/medqa/) (2020; healthcare, knowledge-qa): Context only. Fits nothing agentic; useful as a knowledge floor for a medical assistant and as the source vignettes that AgentClinic and MedCalc-Bench reuse.
- [CORD](https://mahalsystems.ai/aot/evals/benchmarks/cord/) (2019; documents, retail): Fits receipt and expense capture agents. Borrow its two-level label scheme (five superclasses, 42 subclasses) and its line-group annotations as a template for structured extraction targets.
- [Magellan / DeepMatcher entity matching datasets](https://mahalsystems.ai/aot/evals/benchmarks/magellan-er/) (2018; data-analysis): Fits record linkage or deduplication agents that need labeled pairs to test on across domains. Borrow its three flavors (structured, textual, dirty) and its 3:1:1 splits as a template for building a matching test set.
- [Defects4J](https://mahalsystems.ai/aot/evals/benchmarks/defects4j/) (2014; coding): Still the reference set for Java program-repair agents (RepairAgent, ReinFix report on it); borrow the minimized single-commit fix and triggering-test rules for writing repair cases.