{"schema":"aot-evals/site@1","version":"1.0.1","commit":"44af418","generated":"2026-11-04","method":{"title":"The aot-evals method","lead":"This is the reference for what aot-evals does and why: the principles, the measurement frame, the files it keeps in your repository, the commands C0–C19 and their exit gates, the measurements (M), experiments (E), report panels (P) and refusal gates (G) that the skills and the CLI name. The skills read the section they need; you can too.","preface":"**Who it is for.** The method is designed for production, user-facing agents on any stack: their\nwork is specific to the product, so no public benchmark measures it; their volume makes a point of\ncorrectness, cost or latency worth measuring; and their output reaches users without an expert\nchecking it first. Any agent that can be run from a command can use it. The agent is invoked as a\ncommand, so framework, language and model provider do not matter.\n\n**What it is not.** Not trace collection (it reads exports of what your observability already\nrecords), not a model gateway, not a hosted dashboard and not a leaderboard. There is no service\nand no account. Everything it writes is text, diffable and safe to commit; raw traces and PII\nnever enter git. Every reported number carries `n`, `k` and an interval, or states why it cannot\nhave one. When a number cannot be defended, aot-evals refuses to print it and names the gate.","sections":[{"id":"1-principles","title":"1. Principles","markdown":"1. **Cases come from the agent's real traffic and real failures**, not from generic benchmarks.\n Public benchmarks are a source of grading designs, not of cases. Assess matches the agent to\n a local catalog of them (`aot-evals benchmarks`, [references/benchmarks.md](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/references/benchmarks.md))\n without searching; Build may search for newer ones. What the catalog does contribute is the\n **situations** ([references/situations.yaml](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/references/situations.yaml)): the conditions a\n request can put an agent in (unanswerable, ambiguous, forbidden, confidential, multi-turn,\n tool error, …) that benchmarks were built to test and that an agent's own task types never\n name. The discovery report lays the agent's task types (rows, from its traffic) against the\n situations (columns) as a grid of cases that exist against targets; in Build each situation\n is a taxonomy entry, so coverage (C14) reports the grid's empty cells.\n2. **Understand the agent before measuring it**: inventory, infrastructure, volume and usage first.\n3. **Grade outcomes, with code first.** Check what changed in the world. Use a judge only where\n code cannot decide, and only after calibrating it.\n4. **Cheap sources first, raw transcripts second.**\n5. **Keep sensitive data out of version control.** Raw traces stay local and ignored; only\n reviewed, scrubbed cases are committed.\n6. **Record every claim as confirmed or inferred, with its source.** When config and runtime\n disagree, trust the traces.\n7. **Three planes, reported separately, never combined into one score** (§2.1): correctness,\n operational cost, and trust in the instruments.\n8. **Two case sets and two regimes** (§2.2, §2.3). A set built to find errors and a set built to\n estimate quality are different sets. A test that can be re-run and a signal that can only be\n observed in production are different instruments. Every reported number states which set and\n regime it came from.\n9. **Measure the suite itself** (§2.4): coverage against a list you did not derive from your own\n cases, and redundancy by whether cases distinguish versions of the agent.\n\nThe step-by-step process behind principles 1–6 is in\n[references/process/](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/references/process/README.md).","number":1},{"id":"2-measurement-frame","title":"2. Measurement frame","markdown":"### 2.1 Three planes\n\n| Plane | Question | Instrument | Reported as |\n|---|---|---|---|\n| **A. Correctness** | Did the agent do the right thing? | code state-diff checks; a calibrated binary judge where code cannot decide | pass rate per category with CI; `pass^k` |\n| **B. Operational** | What did it cost? | counters, not judgements: $ per run and per successful task, p50/p90 latency, input/output/cached tokens, cache-hit rate, tool calls, retries, errors and timeouts by outcome code, runs killed at a limit | values against per-category budgets, across runs |\n| **C. Instrument trust** | Can plane A be believed? | judge TPR/TNR against human labels; judge self-consistency and cross-family agreement; case-validity screens | its own report panel, with reliability and validity in separate column groups |\n\nPlane B is deterministic. It is recorded on every trial, pass or fail, and needs no ground truth,\njudge or labels, so it is the first thing to instrument. Operational failures (timeouts, rate\nlimits, crashes) are plane-B data: never scored as correctness failures and never dropped.\n\n- **An operational regression is a finding even when correctness is unchanged.** Cost per\n successful task doubling at the same pass rate is a failed comparison (verdict `worse`). A common\n cause is a prompt-prefix edit that invalidates the prompt cache.\n- **A correctness gain that comes with a cost or latency regression beyond budget is a trade**,\n reported with both numbers, not as an improvement (G11).\n\n### 2.2 Two regimes: offline and online\n\n| | Offline (benchmark) | Online (operational) |\n|---|---|---|\n| Subject | frozen cases, in a sandbox or by replay | live traffic |\n| Re-runnable | yes: same inputs, k times | no |\n| Ground truth | the case's expected end state | none; needs a referent such as human labels or user behaviour |\n| Grading | state diff, then a narrow judge | a judge on a sample; behavioural signals; fingerprints on a control chart |\n| Answers | Did this change make the agent better? | Is it getting worse, and where? |\n| Cannot answer | whether the frozen set still resembles production | which of two candidates is better |\n\nNeither substitutes for the other. Offline cases may come from your observability stack, but are\nalways imported as a local export (C0); the suite runs with that stack switched off. aot-evals\nautomates the offline regime today; the online regime (C16, C17) is guidance only.\n\n### 2.3 Two offline case sets\n\n- The **error-finding set** is enriched with hard, ambiguous and rare-but-costly cases. It finds\n bugs and verifies fixes. Its pass rate says nothing about performance on real traffic.\n- The **quality-estimate set** is sampled in proportion to each category's observed share of\n traffic. Its weighted pass rate is the only number the report may call \"quality\".\n\nThe two are disjoint, sized by different arithmetic (C5), and every reported number is labelled\nwith its set.\n\n### 2.4 Coverage and redundancy\n\n**Coverage.** Coverage measured against categories derived from the suite's own cases is always\n100%, so it is measured against a list written for some other purpose: the product spec, the API\nor tool surface, a help-centre topic list, a job-task taxonomy, a regulator's list of duties. The\nreport counts cases per entry on that list, crossed with failure modes, and lists the empty cells\nfirst. A taxonomy entry may declare failure modes `not_applicable`, with a reason: those cells\nleave the empty count but stay visible.\n\n**Redundancy.** A case that every version of the agent passes, or every version fails, cannot tell\nyou which version is better. Over two or more versions, each case is always-pass, always-fail or\ndiscriminating; the report gives the discriminating fraction. Always-pass cases still work as\nregression guards; they just contribute nothing to a comparison. The hardest cases change after a\nprompt rewrite, so the discriminating set is re-derived after every prompt change.","number":2},{"id":"3-repository-contract","title":"3. Repository contract","markdown":"Each agent's suite lives in `evals/agents//`, beside its discovery report. A repository\nwith one agent may keep the suite in `evals/` itself. Pass `--agent NAME` to the CLI when there\nare several. Suites made before that layout live in `evals//`; they are read as they are,\nand `aot-evals migrate` moves them.\n\n```\nevals/agents//\n discovery.md # Assess: the discovery report\n agent.yaml # identity, triggers, tools, model per call site, stable window,\n # and how to invoke it (references/agent-contract.md)\n taxonomy.yaml # the coverage list (§2.4): entries, the source document, the date read\n categories.yaml # trace categories: definition, worth, simple flag, cost matrix, observed share\n failure_modes.yaml # per failure mode: >=1 check, >=1 category, a cost-matrix cell\n budgets.yaml # plane B: cost / latency / token / error-rate budgets per category\n allocation.yaml # C5's plan: sizes, and each unfilled cell scheduled or out of scope\n cases//.json # frozen inputs, start state, trigger, expected end state, set, provenance,\n # requirements (what a correct run must do, each with its checks)\n labels/standard.md # the labelling standard: rules a person applies, with the categories each applies to\n labels/runs.jsonl # C19: whole-run human labels, one line per (run, labeller); no agent output\n labels/rulings.jsonl # §4.2: the verdict on a run its labellers disagreed on\n labels/proposals.yaml # §4.2: rules proposed from rulings, accepted or declined by the owner\n labels/drafts/ # §4.2: cases a grader drafted, waiting for an engineer\n labels/triage.yaml # C19: a cause for each disagreement between the suite and a person\n labels/floors.yaml # optional: C19's floors (min_catch, min_clean, min_runs, min_each)\n checks/\n code/.py # binary state diff: check(start, end, case) -> (bool, detail)\n judge/.md # one failure mode, binary, typed; backend-agnostic (§5)\n judge/.labels.jsonl # human labels, each with the exact judge input labelled\n judge/.judgements.jsonl # cached judge calls, so re-measuring costs nothing\n judge/.calibration.json # C10's measurements\n env/ # fakes, cassettes, seeds, frozen-clock config\n data/ # ignored: exports, labelling pages; only export.manifest.json is committed\n runs//manifest.json # the condition record\n runs//outcomes.jsonl # one line per attempt and per check verdict\n report/report.json # §8\n report/agreement.json # C19: catch and clean pass rates on whole runs, dev and holdout, per category\n report/report-.html # §9\n report/checkpoints/ # §9: one page per owner checkpoint\n journal/ # §9: the activity log; ignores itself\n .gitignore\n```\n\nRules:\n\n- **Paths start at the repository root**, the folder that holds `evals/`: `invoke.cwd` in\n `agent.yaml` and a command judge's `cwd`.\n- **`outcomes.jsonl` is never aggregated at write time.** It has two kinds of line: `kind:\n attempt`, one per (case, trial, attempt), with the outcome code and plane-B counters; and `kind:\n check`, one per (case, check, trial, attempt), with the verdict. Every attempt is written,\n including retries and operational failures. An attempt's check lines are written before its\n attempt line, so a final attempt line on disk means the attempt is complete. `pass^k`,\n flakiness, discrimination and paired tests are computed from these lines.\n- **`manifest.json` is the condition record**: the agent's commit (computed with `evals/`\n excluded, so committing a run or a label is never an agent change) and this suite's commit, every\n model id, harness, fakes, k, total cost, and whether the run was isolated or saw the eval files\n (§4, C12). Counts, cost and observed models are recomputed from `outcomes.jsonl` when the run\n ends. A resumed run lists each resume under `resumed` (`at`, `pairs_done_before`). A run\n started with `--model ` records it as `invoke.model`; the agent receives it as `model`, a\n candidate without `--variable` declares `model=`, and a resume keeps it. Two runs whose manifests differ on any field other than the declared independent\n variable are not compared (G7).\n- **Every case has a `set`**: `error_finding` or `quality_estimate`. There is no default.\n- **Every case has `requirements`**: what a correct run of it must do, each citing a rule of\n `labels/standard.md` and naming the checks that decide it, or why it is unchecked. Every rule\n that applies to the case's category is cited or excused in `rules_not_applicable` (C6, C8).\n- **`taxonomy.yaml` names the document it came from**, and is committed before the first case. A\n model may draft it (C2), but only from a source outside `cases/`.\n- **Unknown values are `null`, never `0`.**","number":3},{"id":"4-commands","title":"4. Commands","markdown":"Each command has an exit gate. A gate is a trust upgrade, not a lock: no command refuses to run\nbecause an earlier gate is unmet. The run manifest records the gates that were unmet when a run\nstarted (`unmet_gates`), and the report gates (§10) grey what each one covers rather than hide\nit. `aot-evals status` says which phase the work is in, how far each category can be trusted,\nwhat changed since the baseline, and the one next step; `aot-evals status --gates` shows every\ngate and the first unmet one.\n\n| # | Command | Does | Exit gate |\n|---|---|---|---|\n| C0 | `import` | Import your data as an export (trace dump, dataset download, CSV/JSONL from your observability stack) into `data/`, with an `export.manifest.json` entry: source, query, window, row count, date. | Every downstream number traces to an export manifest entry; no command calls a live vendor API. |\n| C1 | `inventory` | Write `agent.yaml`: surface, triggers, instructions, tools (credential names only), harness, the model actually used per call site, version history, the stable window, and the invocation command. | Stable window stated with evidence; model per call site observable; `invoke.cwd` exists; the agent runs once end to end from its command. |\n| C2 | `taxonomy` | Build the coverage list from a source outside the suite: the product spec, API/tool surface, help-centre topics or job description. The owner edits it; the source document is recorded. | `taxonomy.yaml` names a source outside `cases/` and a date, and its commit precedes the first case. |\n| C3 | `categorize` | Sort traces into categories, one per trace. Give each a worth and a per-failure-kind cost matrix. Count observed share. | Every category has a definition, an example, worth, a simple flag, a cost row and a measured share. \"Other\" is under 10%. |\n| C4 | `decompose` | Within each category: feature → scenario → failure mode. Mark each failure mode code-checkable, judge-needed or monitor-only, and set plane-B budgets. | Every failure mode names its category, check type and cost cell; `budgets.yaml` covers every category. |\n| C5 | `allocate` | Size both case sets and print the arithmetic. Error-finding: ≥10 per category, 3 for simple ones. Quality-estimate: proportional to observed share, sized by the interval the owner needs (M14). | Both budgets printed with target intervals and the minimum detectable effect (M7); every cell filled, scheduled, or declared out of scope with a reason. |\n| C6 | `author` | Write the labelling standard, then cases: frozen inputs, start state, trigger, scripted follow-ups, expected end state, and requirements walked rule by rule from the standard. Prefer self-verifying pairs where state allows. Scrub before writing. | The standard exists; every case has a set, a category, failure modes, provenance, requirements covering every rule that applies to it, and no secrets. |\n| C7 | `screen` | Per case: a human read; the checks must pass on start → expected end (reference) and fail on start → start (do nothing); a third pass, do-nothing plus a success claim, catches checks that trust the agent's report (G10). With a reference answer: other correct wordings must pass, the reference answer with nothing changed must fail, and an answer with every number wrong is reported if it passes. Drop cases that fail, and record the drop. | Screens recorded per case; drop rate reported. |\n| C8 | `checks` | Implement plane-A checks: code state diffs first, graded against the goal rather than one path; binary judges only where code cannot decide; side-effect checks on every agent; a grounding check for every fact a prose answer states. A check never uses the agent's own report of success. | Every failure mode has at least one binary check; side-effect checks present; every requirement names a check that runs on its case, or why it is unchecked. |\n| C9 | `label` | Collect human labels per judge, stratified across categories: one item per case first, then other trials where the answers differ. Target 100–200; below 60 the interval is too wide to act on. | Label count and stratification printed per judge. |\n| C10 | `calibrate` | Measure each judge against its labels: TPR and TNR separately (never raw agreement), κ, confusion matrix, cost per judgement. Choose a probabilistic backend's threshold on the labels. Version the criteria. | Every judge has TPR/TNR on ≥60 labels at or above its floors, a named backend and model family, and a version. Uncalibrated judges may run but never count toward headline numbers. |\n| C11 | `reliability` | Run *m* fresh sessions per item on two backends from different model families; report agreement at three levels (all / majority / none) next to C10's TPR/TNR. | Self-consistency and cross-family agreement reported next to TPR/TNR, never in place of it. |\n| C12 | `baseline` | Run the suite k = 3–5 times on an unchanged, committed build. Write per-trial outcomes and plane-B counters. Compute the variance band. | Run complete and finished, manifest complete, no eval-file exposure, variance band per category, plane B recorded against `budgets.yaml`. |\n| C19 | `agreement` | Runs after C12 (numbered last so earlier numbers stay stable). A person labels whole baseline runs pass or fail with a note, seeing the request, the answer and what changed, never the checks. Every labelled run is re-graded from its raw record with the suite as it is now, so labels outlive suite versions. Report the catch rate and clean pass rate (M29), dev and holdout; give every dev disagreement a cause. | ≥60 labelled runs with ≥15 of each kind; a measurement current for the suite and the labels; dev catch and clean pass rates at or above their floors (85% by default); every dev disagreement triaged. Until then G14 fails and comparisons are provisional. |\n| C13 | `compare` | Run a candidate on the same cases with one declared independent variable. Report gains, losses and a paired test, never a net delta alone, with plane-B deltas alongside. | Paired test present; gains and losses listed; manifests differ only on the declared variable. |\n| C14 | `coverage` | Count cases per (taxonomy entry × failure mode), zeros included, with the source document named. | Coverage reported against an external list, or refused (G5). |\n| C15 | `redundancy` | Over ≥2 agent versions, classify each case; report the discriminating fraction and a per-case index; propose prunes and cells to extend. | Discriminating fraction published; below the threshold (default 0.3), a prune/extend proposal. |\n| C16 | `drift` | Re-run the frozen set on a schedule and on every prompt, skill, tool, model or harness change, against the baseline's band. | Guidance only; not yet automated. |\n| C17 | `online` | Set up the online regime: shadow → canary → online judging; behavioural signals; fingerprints. | Guidance only; not yet automated. |\n| C18 | `report` | Write `report.json` (§8), then the HTML report (§9). | Every gate's status is in the report. |\n\nC3–C18 repeat as a loop; §4.1 says what each kind of change re-opens. A prompt rewrite\nre-derives the hard-case set, and a shift in category mix re-opens C3 and C5.\n\n**How runs work.**\n\n- The agent is a command: the case goes in as JSON on stdin and one JSON result comes back on\n stdout, with optional `setup`, `seed`, `observe` and `reset` hooks\n ([references/agent-contract.md](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/references/agent-contract.md)). Checks grade what `observe`\n returns, never what the agent says it did.\n- Outcome codes (`ok`, `timeout`, `rate_limited`, `provider_error`, `no_output`, `unparseable`,\n `model_disabled`, `auth_error`, `budget_exceeded`, `cli_missing`) are recorded per attempt. Only\n `ok` is scored; retryable codes are retried up to `max_attempts`.\n- A run stops after `--stop-after` (default 5) cases in a row end in an operational failure; a\n stopped run is never finished. `--pause-every N --pause-seconds S` paces a rate-limited runtime.\n- `aot-evals run --resume RUN_ID` carries on a stopped or interrupted run in the same directory. It\n skips every (case, trial) with a final attempt, including one that ended in an operational\n failure, and runs the rest, appending to the same `outcomes.jsonl`. A pair whose last attempt\n isn't final is run again, with attempt numbers after the ones already used, so a crashed\n attempt's lines never join a new one's. A truncated last line is dropped first, and the `setup`\n and `reset` hooks run before anything else. Resume is refused when the run is finished, or when\n the agent commit, the suite's commit (changes under `runs/` aside), `invoke.env`, a judge's sha,\n the case set, the code checks or the fakes differ from the manifest, or when the agent has\n uncommitted changes: one run never mixes two versions. The run keeps the `purpose`, `k`, case\n set and judges setting it started with. Once finished, it counts as a complete run (C12).\n- **The agent must not see the cases.** `invoke.isolate: true` runs the agent and its hooks from a\n per-run copy of the repository without `evals/` (this suite's `env/` is kept). Every run, isolated\n or not, records eval files changed during the run and agent output that names a case file; either\n keeps the run from being a baseline.\n\n**How comparisons work (C13).** Each side's verdict on a case is its majority over k trials. Gains\nand losses are majority flips, tested with McNemar's exact test; a bootstrap CI covers the mean\nper-case delta. A change is `better` or `worse` only when it is significant and outside the\nbaseline's variance band; otherwise `indistinguishable`. The declared variable is a kind\n(`model`, `prompt`, `config`, `tools`, `harness`, ...) that maps to the manifest fields allowed to\ndiffer. Runs of one build split across several runs (for example category by category, to pace a\nrate-limited runtime) count as one side: their case sets, checks and judges are merged per item.\n\n**How redundancy works (C15).** Versions are runs grouped by the fields that define the build, so\nrepeats of one build never look like two versions. The prune proposal never prunes\nquality-estimate cases, sends always-fail cases to review, keeps regression guards for coverage\ncells and categories, and re-runs the pairwise tests to show the detectable effect is unchanged.\n\n### 4.1 Phases, trust and iteration\n\nThe work has three phases, each ending in something handed to the next group of people:\n\n| Phase | Who | Ends with |\n|---|---|---|\n| **Assess** | the owner and tech lead, with Claude | the discovery report: why the agent is being tested, how it is built, where it can run safely, and the golden test case grid |\n| **Build** | engineering | the loop running: the agent wired to the contract, cases seeded from real runs with a check on each, a baseline, and the grader application started |\n| **Judge** | graders (product owners, experts, end users) | each category validated: a person's verdicts on whole runs, the suite held to them, judges calibrated, rulings folded into the standard |\n\n**The lightest Build** (`aot-evals build`, then `aot-evals build --run`) is the one that gets the loop running: an agent that runs from a command\n(or recorded runs replayed, C1), 10 to 20 cases from real runs in the largest categories (C6), a\nside-effect check and one code check per category (C8), and a baseline (C12). Every other\ncommand is an upgrade that raises trust in a number: provenance (C0), the coverage list (C2),\nshares and the cost matrix (C3), budgets (C4), allocation (C5), screens (C7), judge labels,\ncalibration and reliability (C9 to C11), whole-run agreement (C19), coverage (C14) and\nredundancy (C15). The report shows which upgrades are missing and greys what they cover.\n`aot-evals build` drafts what it can from the discovery report: categories from its task types,\nthe coverage list from its situations (to be committed before the first case), and one failure\nmode on every case checked by a generic code check, `end-state-matches` (the world ends as the\ncase expects), so a first pass needs no check code. Each draft is marked for the owner to confirm.\n\n**Trust per category.** A category is `untested` until a finished baseline covers it,\n`provisional` once it has numbers, and `validated` when the suite agrees with a person on its\nruns: C19's floors met suite-wide, and the category's own dev catch rate and clean pass rate at\nor above the floors (85% by default). `report/agreement.json` carries the per-category rates.\nProvisional numbers print, labelled provisional (G14).\n\n**The baseline** is, for each current case, the newest finished baseline run that covers it: a\nre-run of one category replaces that category's baseline and leaves the others'. The agent has\nchanged only when its files outside `evals/` differ from the baseline's, not merely its commit id.\n\n**What a change invalidates.** `status` compares the suite and the agent with the baseline and\nnames each change with its short path back:\n\n| Change since the baseline | Invalidates | The short path |\n|---|---|---|\n| The agent (commit, model, prompt, tools, harness) | the baseline; after a prompt rewrite, the hard-case set | a candidate run with one declared variable, then `compare`; a person labels only the runs whose verdict changed, and the provisional categories |\n| A new category (a new use case) | nothing else; the new category is untested | checks for its failure modes, a baseline that includes it, labels until it clears the floors |\n| Cases added or changed | nothing until they run | screen them; include them in the next baseline (unscreened cases stay out of gated numbers, G8) |\n| A check changed or added | the measured agreement | re-grade the labelled runs offline (`agreement`); no agent runs |\n| A judge changed (its version) | that judge's calibration | recalibrate from cached judgements, then re-grade |\n| The labelling standard changed (a ruling became a rule) | the labels the rule could flip | recheck those labels, then re-grade |\n| Labels added | the measured agreement | measure it again |\n\nAn uncommitted agent is named too: a run of it can't be compared (G7). When the agent's\ncurrent version has already run as a candidate and been compared, the step is to decide: have\na person label only the runs whose verdict moved, then run it as the new baseline if it ships.\n\n### 4.2 The bench of graders and the grader application\n\n`aot-evals serve` starts the grader application: a local web page, served from the suite, for\nthe Judge phase. It binds to 127.0.0.1 and needs the session token in the link it prints;\n`&labeller=NAME` on that link gives each grader their own. Everything it does writes the suite's\nown files, which are reviewed and committed like any other change:\n\n| What a grader does | Where it is written |\n|---|---|\n| Labels whole runs pass or fail, with a note, blind to the checks, the expected state and other people's labels | `labels/runs.jsonl`, one line per (run, labeller) |\n| Settles a run two labellers disagree on | `labels/rulings.jsonl` |\n| Gives a cause to a disagreement between the suite and a person | `labels/triage.yaml` |\n| Accepts an answer the suite failed as another correct wording | the case's `expected_end_state.output_variants`; the case is screened again |\n| Proposes a rule from a pattern of rulings; the owner accepts or declines it | `labels/proposals.yaml`; an accepted rule joins `labels/standard.md` |\n| Labels items for a judge, and recalibrates it (confirming the cost of any new calls) | `checks/judge/.labels.jsonl`, `.calibration.json` |\n| Drafts a new case in the shape of an existing one | `labels/drafts/.json`, until an engineer finishes it |\n\nRules:\n\n- **Several people label the same run, blind.** Each labeller's latest label is kept. Where all\n of a run's labellers agree, that is its verdict; where they differ, it has no verdict until\n someone rules, and C19 stays open while any such run is unruled. Agreement between each pair of\n labellers (share agreed and Cohen's κ) is reported beside the suite's agreement with people.\n- **Queues are per labeller.** A new labeller starts with up to five runs whose answer is\n settled (a ruling, or two people agreeing) and sees the settled answer after each label. After\n that, one run in four is one someone else labelled, so agreement between people can be measured,\n and the rest are runs nobody has labelled, one per case first. After a change, the queue can be\n limited to the cases whose verdict moved (`aot-evals agreement label queue --changed`).\n- **Re-grading is offline.** The application re-grades the labelled runs from their recorded raw\n records with the suite as it is now: no agent runs and no judge calls.\n- **An accepted wording is screened.** Every accepted wording must pass every check (C7). If a\n check still rejects it, the disagreement is triaged as a check bug for an engineer.\n- **The engineering inbox** lists what only engineering can do: triaged disagreements whose cause\n is a fix to a case or check, drafted cases, and changes that need the agent run.\n\n### 4.3 The components\n\nAn optimization loop around an agent has the same parts whatever the agent does. `aot-evals\ncomponents` lists them, each with the phase that builds it and its state read from the suite\n(done, partly, not yet, not automated yet, or doesn't apply). The discovery report may record a\ncomponent's state before a suite exists (`components:` in its front matter); the suite's own\nreading wins once it says more.\n\n| Area | Components |\n|---|---|\n| Agent eval infrastructure | harness and sandbox runtime; mock services, staging accounts, a safe database; record and replay; seeded, realistic, anonymized data |\n| Golden test cases | cases by category; deterministic checks first; private holdouts |\n| Grader UI | each problem graded like a test, with the reason (`aot-evals serve`) |\n| Bench of graders | a calibration set first; two blind labels per run; graders from customer and expert pools |\n| LLM judges | calibrated on human labels; qualitative dimensions |\n| Synthetic test cases | generated from golden cases, validated by graders |\n| Evaluation engine | golden test and judge evals; change scoping against the test sets; A/B tests; trace monitoring |\n| Objective monitoring | value and traffic weighting; business alignment; distribution change detection |\n| Agent testing and modeling | significance testing; noise floor and k; misclassification analysis; test runner and isolated compute |","number":4},{"id":"5-judges","title":"5. Judges","markdown":"A judge is a check for one failure mode that asks one binary, typed question: a `noul` (a\ntrue/false claim, with `detects` saying whether yes means the failure is present or the\nrequirement is met), a `choice` over declared options, or a `score` on a rubric level. The judge\nlayer is backend-agnostic ([references/judge-backends.md](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/references/judge-backends.md)):\n\n| Backend | Use |\n|---|---|\n| Code | anything decidable from state; always first |\n| System One (`systemone`), such as Jev | typed questions answered with a probability; cheap and fast enough to judge every attempt |\n| Generative (`anthropic`), a different model family from the agent | reference-guided grading, pairwise comparison, anything needing a written critique |\n| `command` | any judge you run yourself: another provider, a gateway, a rules engine |\n\nRules:\n\n1. **Every backend is calibrated in C10.** A typed answer can still be wrong.\n2. **A backend's confidence is not trusted until measured on your labels.** A probabilistic\n backend's threshold is chosen by maximising TPR + TNR on the dev labels; the reported rates come\n from 5-fold out-of-fold predictions, and a hash-assigned 20% holdout is reported separately. The\n floors are `min_tpr` / `min_tnr` (default 0.75).\n3. **A judge counts toward a headline only when it is calibrated on its current version and\n backend.** Stored scores are re-thresholded with the calibrated threshold, so calibrating after a\n run changes the report without re-running anything.\n4. **Labels carry the judge's input.** Each label includes the exact input that was labelled\n (after a secret scan), so calibration can be re-run on any backend without the raw run data.\n5. **Judgements are cached** per backend (the question, the inputs and that backend's own config),\n so adding or changing a backend does not re-pay for the others.\n6. **No silent fallback.** A refusal or an error is an instrument error, never a verdict from\n another model. A rate-limit error stops that backend's calls at once.\n7. **Different family for pairwise grading** (G4).\n\nE5 compares backends on the same labels, by TPR/TNR and cost per judgement, and is how each\njudge's backend is chosen.","number":5},{"id":"6-measurements","title":"6. Measurements","markdown":"`N` = cases, `k` = trials per case, `L` = human labels. Plane refers to §2.1. Measurements marked †\nare guidance for the online regime and are not yet computed by the CLI.\n\n| id | Plane | Measurement | Definition | Use |\n|---|---|---|---|---|\n| M1 | A | Per-check outcome | binary, per (case, check, trial) | the unit every other plane-A number is computed from |\n| M2 | A | Pass rate | mean over trials and cases, per category, with a cluster-robust CI over cases (Wilson when that collapses) | never reported suite-wide without the per-category table |\n| M3 | A | `pass^k` | fraction of cases passing all their trials | reliability across repeats; 75% per trial ≈ 42% at k = 3 |\n| M4 | A | Flakiness | per case: 0 < passes < k | reported as a finding; sets the k budget |\n| M5 | A | Variance band | same build, ≥3 repeats, spread per category | no change inside the band is called drift or improvement |\n| M6 | A | Paired difference | per-case paired delta, McNemar and bootstrap CI, gains and losses listed separately | a net delta alone is not reported |\n| M7 | A | Minimum detectable effect | from N, k and per-case variance at 80% power | printed before an experiment runs |\n| M8 | C | Judge TPR / TNR | against that judge's labels, separately, with CIs | raw agreement is not reported: an always-pass judge agrees 95% of the time at a 95% base rate |\n| M9 | C | Judge reliability | self-consistency across *m* sessions; agreement across two families; three-level agreement | reported next to M8, never in place of it |\n| M10 | C | Label status | L per judge and the resulting interval width | under 60: uncalibrated; 100–200: usable |\n| M11 | A | Check trigger rate | how often each check fires over the report's runs (the baseline by default), and how often it is the only failing check | a check that never fires across several versions, or always fires with another, is a prune candidate. On a baseline that passes everything, every rate is 0 and says nothing about the check |\n| M12 | A | Coverage per cell | cases per (taxonomy entry × failure mode), zeros included, source named | G5 |\n| M13 | A | Discriminating fraction | share of cases not always-pass or always-fail across ≥2 versions | §2.4 |\n| M14 | A | Quality estimate | CI on the share-weighted pass rate of the quality-estimate set, weights published | the only number reported as \"quality\" |\n| M15 | A | Error-finding yield | errors found per case reviewed, vs random selection | judges a case-selection method |\n| M16 | A | Disagreement yield | size of the set where judges or models disagree, and its share of all errors | judges a case-selection method |\n| M17 † | A | Hard-set churn | overlap of the hardest-*x*% set before and after a prompt change | re-derive the hard set when it churns |\n| M18 | B | Cost per successful task | run cost ÷ successes | cost per run is the wrong denominator |\n| M19 | B | Latency | p50/p90; runs killed at a limit | budgeted per category |\n| M20 | B | Tokens and cache | input, output and cached tokens; cache-hit rate | a cost jump at an unchanged pass rate usually means a prompt-prefix edit broke the cache. Compare dollars, not tokens |\n| M21 | B | Errors and retries | count and rate (Wilson interval) per outcome code, over every attempt including retries | needs no judge; the cheapest regression signal |\n| M22 | C | Case validity | reference pass, do-nothing fail, human read; drop rate at C7 | unscreened cases do not count toward gated numbers (G8) |\n| M23 † | C | Fake fidelity | contract test of each fake against the real API | a drifted fake looks the same as a regression |\n| M24 | C | Simulated-user sensitivity | the same suite under ≥2 user simulators, reported per simulator | G9 |\n| M25 | C | Compounded validity | product of stated stage validities (task × simulator × judge) | printed in the report header; 70% × 70% × 70% ≈ 34% |\n| M26 | A | Leakage screen | overlap of case provenance with likely training or benchmark sources | required for cases built from a public source |\n| M27 † | B | Online fingerprints | per agent: unanswered requests, time to first response, tokens and cost per turn, tool calls per turn, category mix, model id per call | drift in the part of the system that cannot be sandboxed |\n| M29 | C | Whole-run agreement | catch rate (share of runs a person fails that the suite fails) and clean pass rate (share a person passes that the suite passes), each with a Wilson interval, dev and holdout; a run with no verdict counts as wrong either way | C19, G14. Per-check validity (M8, M22) does not imply it: a suite can clear every screen and miss what a person fails runs for |\n| M28 † | A | Behavioural signals | post-output edits, acceptance, follow-up corrections, re-asks | free labels for C9 |","number":6},{"id":"7-experiments","title":"7. Experiments","markdown":"No comparison runs before E1. Experiments marked † are guidance only.\n\n| id | Experiment | Design | Produces |\n|---|---|---|---|\n| E1 | Noise floor | same build, same cases, k ≥ 3, ideally on two different days | M5, M4, the k budget |\n| E2 | Candidate comparison | paired on identical cases; one declared independent variable | M6 with gains and losses, M3, plane B |\n| E3 | Judge calibration | stratified labels per judge; iterate; hold out a slice | M8, M10, judge versions |\n| E4 | Judge reliability | *m* sessions × two families, plus a human-labelled subsample | M9 next to M8 |\n| E5 | Judge backend comparison | same judge and labels: System One vs generative vs your own | cost per judgement, TPR/TNR; the backend for each judge |\n| E6 | Downgrade / routing probe | a cheaper model or routing policy against a near-100%-pass regression suite, with the prompt re-optimised first | M6, M18–M20 |\n| E7 † | Selector comparison | random vs uncertainty vs disagreement selection at a fixed label budget | M15, M16 |\n| E8 | Coverage audit | project the suite onto the taxonomy; triage every empty cell | M12, an ordered authoring queue |\n| E9 | Redundancy / prune | classify every case over ≥2 versions; propose prunes | M13, M11, a smaller suite with the same detectable effect |\n| E10 | Weight sensitivity | for composite scores only: vary the weights and check whether the ranking holds | whether the composite is stable |\n| E11 † | Shadow / canary | the candidate states what it would do; compare with what happened; then a fraction of live traffic | the gate before write access, for agents with no sandbox |\n| E12 † | Fake contract test | replay each fake against the live API on a schedule; diff | M23 |\n| E13 † | Drift watch | the frozen set on a schedule, plus M27 on a control chart with M5's limits | the standing check for provider-side and self-inflicted changes |\n| E15 | Suite validity | whole-run human labels on baseline runs, the suite re-graded on them; iterate on dev disagreements by cause; check holdout once; label a fresh batch before a new claim | M29, a triaged list of what the suite misses and why |\n| E14 | Operational sweep | plane B only, no labels or judges: budgets vs actuals per category | M18–M21; can run in week one, from an export |","number":7},{"id":"8-reportjson","title":"8. report.json","markdown":"Written by C18 (`aot-evals report`) to `report/report.json` and read by the HTML report. The\nschema is versioned (`aot-evals/report@1`); keys may be added within a version, never removed or\nrenamed.\n\n```jsonc\n{\n \"schema\": \"aot-evals/report@1\",\n \"generated\": \"2026-01-01T00:00:00Z\",\n \"agent\": { \"id\": \"...\", \"stable_window_since\": \"...\", \"model_ids\": {} },\n \"conditions\": { \"agent_commit\": \"...\", \"evals_commit\": \"...\", \"harness\": \"...\", \"k\": 5,\n \"judge_backends\": [ { \"check\": \"...\", \"backend\": \"...\", \"model\": \"...\",\n \"family\": \"...\", \"version\": \"...\" } ],\n \"simulator\": null, \"cost_usd\": 0.0,\n \"data_exports\": [ { \"source\": \"...\", \"window\": \"...\", \"rows\": 0 } ] },\n \"regime\": \"offline\",\n \"taxonomy\": { \"name\": \"...\", \"source\": \"...|MISSING\", \"read_on\": \"...\", \"entries\": [] },\n \"sets\": { \"error_finding\": { \"n\": 0, \"mde_at_80_power\": 0.0 },\n \"quality_estimate\": { \"n\": 0, \"weights_source\": \"...\", \"estimate\": {} } },\n \"categories\": [ { \"id\": \"...\", \"worth\": \"high\", \"simple\": false,\n \"observed_share\": 0.0, \"n_cases\": 0,\n \"pass_rate\": { \"point\": 0.0, \"ci\": [0, 0], \"n\": 0, \"n_cases\": 0, \"k\": 5,\n \"set\": \"...\", \"ci_method\": \"...\" },\n \"pass_pow_k\": 0.0, \"pass_pow_k_reason\": null, \"flaky\": 0, \"variance_band\": [0, 0],\n \"operational\": { \"cost_per_success\": 0.0, \"cost_per_run\": 0.0,\n \"latency_p50\": 0, \"latency_p90\": 0,\n \"tokens_input\": 0, \"tokens_output\": 0, \"cache_hit_rate\": 0.0,\n \"tool_calls\": 0.0, \"outcomes\": { \"ok\": 0, \"timeout\": 0 },\n \"retries\": 0, \"killed\": 0,\n \"budget\": {}, \"budget_status\": \"within|over\" },\n \"excluded\": null } ],\n \"checks\": [ { \"id\": \"...\", \"type\": \"code|judge\", \"failure_mode\": \"...\",\n \"trigger_rate\": 0.0, \"sole_failure_rate\": 0.0,\n \"judge\": { \"backend\": \"...\", \"model_family\": \"...\", \"agent_model_family\": \"...\",\n \"tpr\": 0.0, \"tnr\": 0.0, \"labels\": 0, \"kappa\": 0.0, \"threshold\": null,\n \"self_consistency\": 0.0, \"cross_family\": 0.0,\n \"confidence_calibration\": { \"ece\": null, \"brier\": null },\n \"cost_per_judgement\": 0.0,\n \"status\": \"calibrated|uncalibrated\", \"status_reasons\": [] } } ],\n \"coverage\": { \"list\": \"external|MISSING\", \"cells\": [ { \"taxonomy_entry\": \"...\", \"failure_mode\": \"...\", \"n\": 0 } ],\n \"empty_cells\": 0, \"authoring_queue\": [] },\n \"redundancy\": { \"candidates\": [], \"always_pass\": 0, \"always_fail\": 0, \"discriminating\": 0,\n \"discriminating_fraction\": 0.0, \"per_case\": [], \"proposal\": {} },\n \"comparison\": { \"baseline\": \"...\", \"candidate\": \"...\", \"paired\": true, \"refused\": null,\n \"gains\": [], \"losses\": [], \"net\": 0.0,\n \"test\": { \"name\": \"mcnemar_exact\", \"p\": 0.0, \"ci\": [0, 0] },\n \"mde_at_80_power\": 0.0, \"per_set\": {},\n \"operational_delta\": { \"cost_per_success\": 0.0, \"latency_p90\": 0, \"error_rate\": 0.0 },\n \"verdict\": \"better|worse|indistinguishable|trade\", \"verdict_reasons\": [] },\n \"validity\": { \"stages\": { \"task\": null, \"simulator\": null, \"judge\": null }, \"compounded\": null },\n \"gates\": [ { \"id\": \"G5\", \"status\": \"pass|fail|not_applicable\", \"detail\": \"...\", \"blocks\": [] } ],\n \"next_actions\": [],\n \"cases\": [ { \"id\": \"...\", \"set\": \"...\", \"category\": \"...\", \"provenance\": \"...\",\n \"screen\": { \"human_read\": true, \"reference_passes\": true, \"donothing_fails\": true },\n \"trials\": [ { \"run\": \"...\", \"trial\": 0, \"attempt\": 0, \"outcome\": \"ok\", \"passed\": false,\n \"checks\": [ { \"id\": \"...\", \"type\": \"code\", \"passed\": false, \"detail\": \"...\",\n \"counted\": true } ],\n \"operational\": { \"cost_usd\": 0.0, \"latency_ms\": 0,\n \"tokens_input\": 0, \"tokens_output\": 0,\n \"tool_calls\": 0, \"retries\": 0 },\n \"state_diff\": \"...\", \"trace_ref\": \"...\" } ] } ]\n}\n```\n\nRules: unknown values are `null`, never `0`. A missing coverage list is the string `\"MISSING\"`,\nnot an omitted key. Nothing is pre-rounded. There is one category row per (category, set),\nlabelled by `pass_rate.set`. Per-trial outcomes and plane-B counters are kept so the renderer can\nrecompute anything. `trace_ref` is an id or a local path, never the trace.","number":8},{"id":"9-the-html-report-checkpoints-and-the-activity-log","title":"9. The HTML report, checkpoints and the activity log","markdown":"aot-evals ships the schema (§8), the panel list and the rules below, not a dashboard. On request,\nClaude writes one self-contained local HTML file: data inlined as JSON, no network requests, no\nCDN, no build step. It opens from the filesystem and can be attached to a pull request or an email.\nEach file is written for a question: a report on \"why did the candidate regress\" leads with the\ncomparison and the losing cases; one on \"is this suite any good\" leads with gates, coverage and\ndiscrimination. `aot-evals report --html` writes the branded shell, and Claude writes the panels\ninto it (the `report` skill).\n\n| id | Panel | Shows |\n|---|---|---|\n| P0 | Conditions | agent and commits, every model id, harness, k, N per set, regime, data exports, date, total cost |\n| P1 | Gates | every gate in §10, pass/fail, and what each failure blocks; above the results |\n| P2 | Validity | task × simulator × judge and the compounded product (M25) |\n| P3 | Correctness by category | pass rate with CI and `pass^k`, ordered by worth; variance band shaded; n, k and set on every row |\n| P4 | Operational by category | plane B against `budgets.yaml`; over-budget cells flagged. Next to P3, never merged with it |\n| P5 | Coverage | taxonomy entry × failure mode counts; empty cells outlined; source in the caption |\n| P6 | Discrimination | always-pass / always-fail / discriminating, with counts and the discriminating fraction |\n| P7 | Comparison | gains and losses as separate bars, each changed case linked; paired test and MDE in the caption; plane-B deltas beside it. No net-only view |\n| P8 | Judges | per judge: backend and family vs the agent's, confusion matrix, TPR/TNR with CIs, labels, κ, threshold, cost; reliability in a separate column group. Uncalibrated judges greyed |\n| P9 | Flakiness | per case, k marks in run order: pass, fail, operational failure |\n| P10 † | Drift | the frozen set over time against M5's band |\n| P11 † | Online | fingerprints on control charts, sampling rate, referent |\n| P12 | Case detail | inputs, start state, expected end state, actual state diff, each check's verdict, plane-B counters, trace reference, provenance, screen status |\n| P13 | Next actions | empty coverage cells, judges under 60 labels, categories under their k budget, prune candidates, over-budget categories; each with the command that clears it |\n\nRules:\n\n1. Every number shows `n`, `k` and an interval, or says why no interval can be computed.\n2. \"Quality\" labels only a quality-estimate-set number with published weights. Error-finding\n numbers are labelled \"error-finding set — not a quality estimate\".\n3. Correctness and operational numbers are never combined, and an operational regression is shown\n even when correctness is unchanged.\n4. Anything below its gate is shown greyed and not used as a headline; it is never hidden.\n5. Reliability and validity never share a column group.\n6. The headline is the per-category table, not a single aggregate pass rate.\n7. Every claim in prose links to the panel it comes from.\n\n**Checkpoints and the activity log.** The work is paced by pages, so the owner can follow it in a\nbrowser and steer at every gate.\n\n- **Activity log.** `journal/activity.jsonl` is append-only: every CLI command writes an entry\n (command, gate, exit code, duration, refusal message), and the skills add their steps, findings,\n decisions and questions with `aot-evals log`. Each append re-renders `activity.html`, which\n reloads itself and says when Claude is waiting. The journal ignores itself in git: it may quote\n unreviewed trace text, and the committed record of a decision is the file it changed.\n- **Checkpoints.** `aot-evals checkpoint` writes a page for one gate: the C0–C19 roadmap in plain\n words, Claude's summary, the log since the last checkpoint, every design file rendered as tables,\n the owner's questions, what still blocks the gate, and the next step. It makes no model calls.\n- **Pacing.** Every skill works until a gate clears or the work needs the owner, publishes a\n checkpoint, and stops until the person replies.\n- **Telemetry.** `start`, `discovery-page`, `checkpoint`, `report` and `demo` each send one\n pseudonymous milestone event when they succeed (install id, event, gate id, version, OS, time;\n nothing about the agent or its data). It is off whenever Claude Code's own telemetry is off,\n on cloud providers and in CI; `aot-evals telemetry` shows, shows the payload of, and switches\n it. Details: [SECURITY.md](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/SECURITY.md#telemetry).\n\nVisual style follows [brand/README.md](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/brand/README.md).","number":9},{"id":"10-gates","title":"10. Gates","markdown":"| id | Condition | Refuses |\n|---|---|---|\n| G1 | No condition record (model ids, commits, harness, k) | the whole report |\n| G2 | `k = 1` | any claim of improvement, regression or drift |\n| G3 | Judge uncalibrated (under 60 labels, below its floors), or reported as raw agreement | that judge's contribution to any headline; the judge is greyed |\n| G4 | Generative judge from the same model family as the agent, for pairwise or preference grading | the pairwise result. A System One judge satisfies this |\n| G5 | Coverage list with no external source, derived from the suite's own cases, or committed after them | the word \"coverage\"; prints `MISSING` and the reason |\n| G6 | Headline pass rate from the error-finding set | the word \"quality\"; offers the weighted estimate instead |\n| G7 | Comparison without a paired test, or with undeclared manifest differences | the comparison |\n| G8 | Cases that have not passed the reference / do-nothing screen | their inclusion in any gated number |\n| G9 | Simulated user present, results not reported per simulator | the aggregate score |\n| G10 | A check that relies on the agent's own report of success | the check |\n| G11 | A \"better\" verdict while plane B regressed beyond budget | the verdict; it becomes \"trade\", with both numbers |\n| G12 | A number whose data does not trace to an export manifest entry or a committed case | that number |\n| G13 | Retried attempts or operational failures missing from `outcomes.jsonl`, or scored as correctness | the run's plane-A and plane-B numbers |\n| G14 | The suite's verdict on whole runs not measured against a person's, stale, below its floors, or with untriaged dev disagreements (C19) | the word \"validated\"; every comparison is marked provisional |","number":10}]},"principles":[{"number":1,"title":"Cases come from the agent's real traffic and real failures","markdown":"**Cases come from the agent's real traffic and real failures**, not from generic benchmarks. Public benchmarks are a source of grading designs, not of cases. Assess matches the agent to a local catalog of them (`aot-evals benchmarks`, [references/benchmarks.md](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/references/benchmarks.md)) without searching; Build may search for newer ones. What the catalog does contribute is the **situations** ([references/situations.yaml](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/references/situations.yaml)): the conditions a request can put an agent in (unanswerable, ambiguous, forbidden, confidential, multi-turn, tool error, …) that benchmarks were built to test and that an agent's own task types never name. The discovery report lays the agent's task types (rows, from its traffic) against the situations (columns) as a grid of cases that exist against targets; in Build each situation is a taxonomy entry, so coverage (C14) reports the grid's empty cells."},{"number":2,"title":"Understand the agent before measuring it","markdown":"**Understand the agent before measuring it**: inventory, infrastructure, volume and usage first."},{"number":3,"title":"Grade outcomes, with code first","markdown":"**Grade outcomes, with code first.** Check what changed in the world. Use a judge only where code cannot decide, and only after calibrating it."},{"number":4,"title":"Cheap sources first, raw transcripts second","markdown":"**Cheap sources first, raw transcripts second.**"},{"number":5,"title":"Keep sensitive data out of version control","markdown":"**Keep sensitive data out of version control.** Raw traces stay local and ignored; only reviewed, scrubbed cases are committed."},{"number":6,"title":"Record every claim as confirmed or inferred, with its source","markdown":"**Record every claim as confirmed or inferred, with its source.** When config and runtime disagree, trust the traces."},{"number":7,"title":"Three planes, reported separately, never combined into one score","markdown":"**Three planes, reported separately, never combined into one score** (§2.1): correctness, operational cost, and trust in the instruments."},{"number":8,"title":"Two case sets and two regimes","markdown":"**Two case sets and two regimes** (§2.2, §2.3). A set built to find errors and a set built to estimate quality are different sets. A test that can be re-run and a signal that can only be observed in production are different instruments. Every reported number states which set and regime it came from."},{"number":9,"title":"Measure the suite itself","markdown":"**Measure the suite itself** (§2.4): coverage against a list you did not derive from your own cases, and redundancy by whether cases distinguish versions of the agent."}],"planes":[{"plane":"Correctness","question":"Did the agent do the right thing?","instrument":"code state-diff checks; a calibrated binary judge where code cannot decide","reported_as":"pass rate per category with CI; `pass^k`","id":"A"},{"plane":"Operational","question":"What did it cost?","instrument":"counters, not judgements: $ per run and per successful task, p50/p90 latency, input/output/cached tokens, cache-hit rate, tool calls, retries, errors and timeouts by outcome code, runs killed at a limit","reported_as":"values against per-category budgets, across runs","id":"B"},{"plane":"Instrument trust","question":"Can plane A be believed?","instrument":"judge TPR/TNR against human labels; judge self-consistency and cross-family agreement; case-validity screens","reported_as":"its own report panel, with reliability and validity in separate column groups","id":"C"}],"commands":[{"id":"C0","command":"import","does":"Import your data as an export (trace dump, dataset download, CSV/JSONL from your observability stack) into `data/`, with an `export.manifest.json` entry: source, query, window, row count, date.","exit_gate":"Every downstream number traces to an export manifest entry; no command calls a live vendor API.","guidance_only":false},{"id":"C1","command":"inventory","does":"Write `agent.yaml`: surface, triggers, instructions, tools (credential names only), harness, the model actually used per call site, version history, the stable window, and the invocation command.","exit_gate":"Stable window stated with evidence; model per call site observable; `invoke.cwd` exists; the agent runs once end to end from its command.","guidance_only":false},{"id":"C2","command":"taxonomy","does":"Build the coverage list from a source outside the suite: the product spec, API/tool surface, help-centre topics or job description. The owner edits it; the source document is recorded.","exit_gate":"`taxonomy.yaml` names a source outside `cases/` and a date, and its commit precedes the first case.","guidance_only":false},{"id":"C3","command":"categorize","does":"Sort traces into categories, one per trace. Give each a worth and a per-failure-kind cost matrix. Count observed share.","exit_gate":"Every category has a definition, an example, worth, a simple flag, a cost row and a measured share. \"Other\" is under 10%.","guidance_only":false},{"id":"C4","command":"decompose","does":"Within each category: feature → scenario → failure mode. Mark each failure mode code-checkable, judge-needed or monitor-only, and set plane-B budgets.","exit_gate":"Every failure mode names its category, check type and cost cell; `budgets.yaml` covers every category.","guidance_only":false},{"id":"C5","command":"allocate","does":"Size both case sets and print the arithmetic. Error-finding: ≥10 per category, 3 for simple ones. Quality-estimate: proportional to observed share, sized by the interval the owner needs (M14).","exit_gate":"Both budgets printed with target intervals and the minimum detectable effect (M7); every cell filled, scheduled, or declared out of scope with a reason.","guidance_only":false},{"id":"C6","command":"author","does":"Write the labelling standard, then cases: frozen inputs, start state, trigger, scripted follow-ups, expected end state, and requirements walked rule by rule from the standard. Prefer self-verifying pairs where state allows. Scrub before writing.","exit_gate":"The standard exists; every case has a set, a category, failure modes, provenance, requirements covering every rule that applies to it, and no secrets.","guidance_only":false},{"id":"C7","command":"screen","does":"Per case: a human read; the checks must pass on start → expected end (reference) and fail on start → start (do nothing); a third pass, do-nothing plus a success claim, catches checks that trust the agent's report (G10). With a reference answer: other correct wordings must pass, the reference answer with nothing changed must fail, and an answer with every number wrong is reported if it passes. Drop cases that fail, and record the drop.","exit_gate":"Screens recorded per case; drop rate reported.","guidance_only":false},{"id":"C8","command":"checks","does":"Implement plane-A checks: code state diffs first, graded against the goal rather than one path; binary judges only where code cannot decide; side-effect checks on every agent; a grounding check for every fact a prose answer states. A check never uses the agent's own report of success.","exit_gate":"Every failure mode has at least one binary check; side-effect checks present; every requirement names a check that runs on its case, or why it is unchecked.","guidance_only":false},{"id":"C9","command":"label","does":"Collect human labels per judge, stratified across categories: one item per case first, then other trials where the answers differ. Target 100–200; below 60 the interval is too wide to act on.","exit_gate":"Label count and stratification printed per judge.","guidance_only":false},{"id":"C10","command":"calibrate","does":"Measure each judge against its labels: TPR and TNR separately (never raw agreement), κ, confusion matrix, cost per judgement. Choose a probabilistic backend's threshold on the labels. Version the criteria.","exit_gate":"Every judge has TPR/TNR on ≥60 labels at or above its floors, a named backend and model family, and a version. Uncalibrated judges may run but never count toward headline numbers.","guidance_only":false},{"id":"C11","command":"reliability","does":"Run *m* fresh sessions per item on two backends from different model families; report agreement at three levels (all / majority / none) next to C10's TPR/TNR.","exit_gate":"Self-consistency and cross-family agreement reported next to TPR/TNR, never in place of it.","guidance_only":false},{"id":"C12","command":"baseline","does":"Run the suite k = 3–5 times on an unchanged, committed build. Write per-trial outcomes and plane-B counters. Compute the variance band.","exit_gate":"Run complete and finished, manifest complete, no eval-file exposure, variance band per category, plane B recorded against `budgets.yaml`.","guidance_only":false},{"id":"C19","command":"agreement","does":"Runs after C12 (numbered last so earlier numbers stay stable). A person labels whole baseline runs pass or fail with a note, seeing the request, the answer and what changed, never the checks. Every labelled run is re-graded from its raw record with the suite as it is now, so labels outlive suite versions. Report the catch rate and clean pass rate (M29), dev and holdout; give every dev disagreement a cause.","exit_gate":"≥60 labelled runs with ≥15 of each kind; a measurement current for the suite and the labels; dev catch and clean pass rates at or above their floors (85% by default); every dev disagreement triaged. Until then G14 fails and comparisons are provisional.","guidance_only":false},{"id":"C13","command":"compare","does":"Run a candidate on the same cases with one declared independent variable. Report gains, losses and a paired test, never a net delta alone, with plane-B deltas alongside.","exit_gate":"Paired test present; gains and losses listed; manifests differ only on the declared variable.","guidance_only":false},{"id":"C14","command":"coverage","does":"Count cases per (taxonomy entry × failure mode), zeros included, with the source document named.","exit_gate":"Coverage reported against an external list, or refused (G5).","guidance_only":false},{"id":"C15","command":"redundancy","does":"Over ≥2 agent versions, classify each case; report the discriminating fraction and a per-case index; propose prunes and cells to extend.","exit_gate":"Discriminating fraction published; below the threshold (default 0.3), a prune/extend proposal.","guidance_only":false},{"id":"C16","command":"drift","does":"Re-run the frozen set on a schedule and on every prompt, skill, tool, model or harness change, against the baseline's band.","exit_gate":"Guidance only; not yet automated.","guidance_only":true},{"id":"C17","command":"online","does":"Set up the online regime: shadow → canary → online judging; behavioural signals; fingerprints.","exit_gate":"Guidance only; not yet automated.","guidance_only":true},{"id":"C18","command":"report","does":"Write `report.json` (§8), then the HTML report (§9).","exit_gate":"Every gate's status is in the report.","guidance_only":false}],"judge_backends":[{"backend":"Code","use":"anything decidable from state; always first"},{"backend":"System One (`systemone`), such as Jev","use":"typed questions answered with a probability; cheap and fast enough to judge every attempt"},{"backend":"Generative (`anthropic`), a different model family from the agent","use":"reference-guided grading, pairwise comparison, anything needing a written critique"},{"backend":"`command`","use":"any judge you run yourself: another provider, a gateway, a rules engine"}],"measurements":[{"id":"M1","plane":"A","definition":"binary, per (case, check, trial)","use":"the unit every other plane-A number is computed from","guidance_only":false,"name":"Per-check outcome"},{"id":"M2","plane":"A","definition":"mean over trials and cases, per category, with a cluster-robust CI over cases (Wilson when that collapses)","use":"never reported suite-wide without the per-category table","guidance_only":false,"name":"Pass rate"},{"id":"M3","plane":"A","definition":"fraction of cases passing all their trials","use":"reliability across repeats; 75% per trial ≈ 42% at k = 3","guidance_only":false,"name":"`pass^k`"},{"id":"M4","plane":"A","definition":"per case: 0 < passes < k","use":"reported as a finding; sets the k budget","guidance_only":false,"name":"Flakiness"},{"id":"M5","plane":"A","definition":"same build, ≥3 repeats, spread per category","use":"no change inside the band is called drift or improvement","guidance_only":false,"name":"Variance band"},{"id":"M6","plane":"A","definition":"per-case paired delta, McNemar and bootstrap CI, gains and losses listed separately","use":"a net delta alone is not reported","guidance_only":false,"name":"Paired difference"},{"id":"M7","plane":"A","definition":"from N, k and per-case variance at 80% power","use":"printed before an experiment runs","guidance_only":false,"name":"Minimum detectable effect"},{"id":"M8","plane":"C","definition":"against that judge's labels, separately, with CIs","use":"raw agreement is not reported: an always-pass judge agrees 95% of the time at a 95% base rate","guidance_only":false,"name":"Judge TPR / TNR"},{"id":"M9","plane":"C","definition":"self-consistency across *m* sessions; agreement across two families; three-level agreement","use":"reported next to M8, never in place of it","guidance_only":false,"name":"Judge reliability"},{"id":"M10","plane":"C","definition":"L per judge and the resulting interval width","use":"under 60: uncalibrated; 100–200: usable","guidance_only":false,"name":"Label status"},{"id":"M11","plane":"A","definition":"how often each check fires over the report's runs (the baseline by default), and how often it is the only failing check","use":"a check that never fires across several versions, or always fires with another, is a prune candidate. On a baseline that passes everything, every rate is 0 and says nothing about the check","guidance_only":false,"name":"Check trigger rate"},{"id":"M12","plane":"A","definition":"cases per (taxonomy entry × failure mode), zeros included, source named","use":"G5","guidance_only":false,"name":"Coverage per cell"},{"id":"M13","plane":"A","definition":"share of cases not always-pass or always-fail across ≥2 versions","use":"§2.4","guidance_only":false,"name":"Discriminating fraction"},{"id":"M14","plane":"A","definition":"CI on the share-weighted pass rate of the quality-estimate set, weights published","use":"the only number reported as \"quality\"","guidance_only":false,"name":"Quality estimate"},{"id":"M15","plane":"A","definition":"errors found per case reviewed, vs random selection","use":"judges a case-selection method","guidance_only":false,"name":"Error-finding yield"},{"id":"M16","plane":"A","definition":"size of the set where judges or models disagree, and its share of all errors","use":"judges a case-selection method","guidance_only":false,"name":"Disagreement yield"},{"id":"M17","plane":"A","definition":"overlap of the hardest-*x*% set before and after a prompt change","use":"re-derive the hard set when it churns","guidance_only":true,"name":"Hard-set churn"},{"id":"M18","plane":"B","definition":"run cost ÷ successes","use":"cost per run is the wrong denominator","guidance_only":false,"name":"Cost per successful task"},{"id":"M19","plane":"B","definition":"p50/p90; runs killed at a limit","use":"budgeted per category","guidance_only":false,"name":"Latency"},{"id":"M20","plane":"B","definition":"input, output and cached tokens; cache-hit rate","use":"a cost jump at an unchanged pass rate usually means a prompt-prefix edit broke the cache. Compare dollars, not tokens","guidance_only":false,"name":"Tokens and cache"},{"id":"M21","plane":"B","definition":"count and rate (Wilson interval) per outcome code, over every attempt including retries","use":"needs no judge; the cheapest regression signal","guidance_only":false,"name":"Errors and retries"},{"id":"M22","plane":"C","definition":"reference pass, do-nothing fail, human read; drop rate at C7","use":"unscreened cases do not count toward gated numbers (G8)","guidance_only":false,"name":"Case validity"},{"id":"M23","plane":"C","definition":"contract test of each fake against the real API","use":"a drifted fake looks the same as a regression","guidance_only":true,"name":"Fake fidelity"},{"id":"M24","plane":"C","definition":"the same suite under ≥2 user simulators, reported per simulator","use":"G9","guidance_only":false,"name":"Simulated-user sensitivity"},{"id":"M25","plane":"C","definition":"product of stated stage validities (task × simulator × judge)","use":"printed in the report header; 70% × 70% × 70% ≈ 34%","guidance_only":false,"name":"Compounded validity"},{"id":"M26","plane":"A","definition":"overlap of case provenance with likely training or benchmark sources","use":"required for cases built from a public source","guidance_only":false,"name":"Leakage screen"},{"id":"M27","plane":"B","definition":"per agent: unanswered requests, time to first response, tokens and cost per turn, tool calls per turn, category mix, model id per call","use":"drift in the part of the system that cannot be sandboxed","guidance_only":true,"name":"Online fingerprints"},{"id":"M29","plane":"C","definition":"catch rate (share of runs a person fails that the suite fails) and clean pass rate (share a person passes that the suite passes), each with a Wilson interval, dev and holdout; a run with no verdict counts as wrong either way","use":"C19, G14. Per-check validity (M8, M22) does not imply it: a suite can clear every screen and miss what a person fails runs for","guidance_only":false,"name":"Whole-run agreement"},{"id":"M28","plane":"A","definition":"post-output edits, acceptance, follow-up corrections, re-asks","use":"free labels for C9","guidance_only":true,"name":"Behavioural signals"}],"experiments":[{"id":"E1","design":"same build, same cases, k ≥ 3, ideally on two different days","produces":"M5, M4, the k budget","guidance_only":false,"name":"Noise floor"},{"id":"E2","design":"paired on identical cases; one declared independent variable","produces":"M6 with gains and losses, M3, plane B","guidance_only":false,"name":"Candidate comparison"},{"id":"E3","design":"stratified labels per judge; iterate; hold out a slice","produces":"M8, M10, judge versions","guidance_only":false,"name":"Judge calibration"},{"id":"E4","design":"*m* sessions × two families, plus a human-labelled subsample","produces":"M9 next to M8","guidance_only":false,"name":"Judge reliability"},{"id":"E5","design":"same judge and labels: System One vs generative vs your own","produces":"cost per judgement, TPR/TNR; the backend for each judge","guidance_only":false,"name":"Judge backend comparison"},{"id":"E6","design":"a cheaper model or routing policy against a near-100%-pass regression suite, with the prompt re-optimised first","produces":"M6, M18–M20","guidance_only":false,"name":"Downgrade / routing probe"},{"id":"E7","design":"random vs uncertainty vs disagreement selection at a fixed label budget","produces":"M15, M16","guidance_only":true,"name":"Selector comparison"},{"id":"E8","design":"project the suite onto the taxonomy; triage every empty cell","produces":"M12, an ordered authoring queue","guidance_only":false,"name":"Coverage audit"},{"id":"E9","design":"classify every case over ≥2 versions; propose prunes","produces":"M13, M11, a smaller suite with the same detectable effect","guidance_only":false,"name":"Redundancy / prune"},{"id":"E10","design":"for composite scores only: vary the weights and check whether the ranking holds","produces":"whether the composite is stable","guidance_only":false,"name":"Weight sensitivity"},{"id":"E11","design":"the candidate states what it would do; compare with what happened; then a fraction of live traffic","produces":"the gate before write access, for agents with no sandbox","guidance_only":true,"name":"Shadow / canary"},{"id":"E12","design":"replay each fake against the live API on a schedule; diff","produces":"M23","guidance_only":true,"name":"Fake contract test"},{"id":"E13","design":"the frozen set on a schedule, plus M27 on a control chart with M5's limits","produces":"the standing check for provider-side and self-inflicted changes","guidance_only":true,"name":"Drift watch"},{"id":"E15","design":"whole-run human labels on baseline runs, the suite re-graded on them; iterate on dev disagreements by cause; check holdout once; label a fresh batch before a new claim","produces":"M29, a triaged list of what the suite misses and why","guidance_only":false,"name":"Suite validity"},{"id":"E14","design":"plane B only, no labels or judges: budgets vs actuals per category","produces":"M18–M21; can run in week one, from an export","guidance_only":false,"name":"Operational sweep"}],"panels":[{"id":"P0","shows":"agent and commits, every model id, harness, k, N per set, regime, data exports, date, total cost","guidance_only":false,"name":"Conditions"},{"id":"P1","shows":"every gate in §10, pass/fail, and what each failure blocks; above the results","guidance_only":false,"name":"Gates"},{"id":"P2","shows":"task × simulator × judge and the compounded product (M25)","guidance_only":false,"name":"Validity"},{"id":"P3","shows":"pass rate with CI and `pass^k`, ordered by worth; variance band shaded; n, k and set on every row","guidance_only":false,"name":"Correctness by category"},{"id":"P4","shows":"plane B against `budgets.yaml`; over-budget cells flagged. Next to P3, never merged with it","guidance_only":false,"name":"Operational by category"},{"id":"P5","shows":"taxonomy entry × failure mode counts; empty cells outlined; source in the caption","guidance_only":false,"name":"Coverage"},{"id":"P6","shows":"always-pass / always-fail / discriminating, with counts and the discriminating fraction","guidance_only":false,"name":"Discrimination"},{"id":"P7","shows":"gains and losses as separate bars, each changed case linked; paired test and MDE in the caption; plane-B deltas beside it. No net-only view","guidance_only":false,"name":"Comparison"},{"id":"P8","shows":"per judge: backend and family vs the agent's, confusion matrix, TPR/TNR with CIs, labels, κ, threshold, cost; reliability in a separate column group. Uncalibrated judges greyed","guidance_only":false,"name":"Judges"},{"id":"P9","shows":"per case, k marks in run order: pass, fail, operational failure","guidance_only":false,"name":"Flakiness"},{"id":"P10","shows":"the frozen set over time against M5's band","guidance_only":true,"name":"Drift"},{"id":"P11","shows":"fingerprints on control charts, sampling rate, referent","guidance_only":true,"name":"Online"},{"id":"P12","shows":"inputs, start state, expected end state, actual state diff, each check's verdict, plane-B counters, trace reference, provenance, screen status","guidance_only":false,"name":"Case detail"},{"id":"P13","shows":"empty coverage cells, judges under 60 labels, categories under their k budget, prune candidates, over-budget categories; each with the command that clears it","guidance_only":false,"name":"Next actions"}],"gates":[{"id":"G1","condition":"No condition record (model ids, commits, harness, k)","refuses":"the whole report","guidance_only":false},{"id":"G2","condition":"`k = 1`","refuses":"any claim of improvement, regression or drift","guidance_only":false},{"id":"G3","condition":"Judge uncalibrated (under 60 labels, below its floors), or reported as raw agreement","refuses":"that judge's contribution to any headline; the judge is greyed","guidance_only":false},{"id":"G4","condition":"Generative judge from the same model family as the agent, for pairwise or preference grading","refuses":"the pairwise result. A System One judge satisfies this","guidance_only":false},{"id":"G5","condition":"Coverage list with no external source, derived from the suite's own cases, or committed after them","refuses":"the word \"coverage\"; prints `MISSING` and the reason","guidance_only":false},{"id":"G6","condition":"Headline pass rate from the error-finding set","refuses":"the word \"quality\"; offers the weighted estimate instead","guidance_only":false},{"id":"G7","condition":"Comparison without a paired test, or with undeclared manifest differences","refuses":"the comparison","guidance_only":false},{"id":"G8","condition":"Cases that have not passed the reference / do-nothing screen","refuses":"their inclusion in any gated number","guidance_only":false},{"id":"G9","condition":"Simulated user present, results not reported per simulator","refuses":"the aggregate score","guidance_only":false},{"id":"G10","condition":"A check that relies on the agent's own report of success","refuses":"the check","guidance_only":false},{"id":"G11","condition":"A \"better\" verdict while plane B regressed beyond budget","refuses":"the verdict; it becomes \"trade\", with both numbers","guidance_only":false},{"id":"G12","condition":"A number whose data does not trace to an export manifest entry or a committed case","refuses":"that number","guidance_only":false},{"id":"G13","condition":"Retried attempts or operational failures missing from `outcomes.jsonl`, or scored as correctness","refuses":"the run's plane-A and plane-B numbers","guidance_only":false},{"id":"G14","condition":"The suite's verdict on whole runs not measured against a person's, stale, below its floors, or with untriaged dev disagreements (C19)","refuses":"the word \"validated\"; every comparison is marked provisional","guidance_only":false}],"process":[{"slug":"overview","title":"The process","markdown":"How to go from agents running in production with no evals to a first eval\nsuite for each of them. Deployment-agnostic: this folder can be copied\nelsewhere unchanged.\n\nEvery agent gets **at least 10 examples per category** (3 for simple\ncategories): the reviewed traces become its first eval cases and the labels\nits judges are checked against.\n\n| File | Covers |\n|---|---|\n| [principles.md](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/references/process/principles.md) | The six rules the steps follow |\n| [phase-1-understand-the-system.md](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/references/process/phase-1-understand-the-system.md) | Steps 1–4: inventory the agents (with version history and stable window), map infrastructure and traces, measure volume, catalog existing evals |\n| [phase-2-understand-the-usage.md](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/references/process/phase-2-understand-the-usage.md) | Steps 5–7: characterize usage from cheap sources, survey public benchmarks, pull raw traces |\n| [phase-3-build-the-evals.md](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/references/process/phase-3-build-the-evals.md) | Steps 8–12: define trace categories, review at least 10 traces per category, analyze errors into checks and cases, calibrate judges, baseline |\n| [agent-properties-framework.md](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/references/process/agent-properties-framework.md) | Six questions about an agent (verifiable? happy with performance → cost or latency? what is drift? sandbox possible? if not? subjective → what criteria?) and the decision procedure that picks its eval strategy |"},{"slug":"agent-properties-framework","title":"Choosing an eval strategy from an agent's properties","markdown":"Not every agent needs the same kind of eval. Six questions about the agent\ndecide what to build. Answer them per agent (and often per category within an\nagent), then follow the decision procedure at the end.\n\nThe \"What the research adds\" notes below cite\n[eval-best-practices](https://github.com/Mahalanobis-Systems/aot-evals/blob/aot-evals--v1.0.1/references/eval-best-practices.md).\n\n---\n\n## The six questions\n\n### Q1. Is the outcome verifiable?\n\nCan code decide, from the end state, whether the job was done right?\n\n| Answer | Meaning | Eval shape |\n|---|---|---|\n| **Fully** | The right result is a definite state: a row, an event, an email to X, a file with fields F, a number | State-diff checks. No judge. Grade the end state, not the transcript |\n| **Partly** | Some of it is definite (the record exists, the format holds, nothing else changed) and some is judgement (was the summary faithful) | Code checks for the definite part; a narrow judge per subjective part; never a single overall score |\n| **Not** | Correctness is a matter of judgement or taste | See Q6 |\n\nRules of thumb:\n- **Side effects outside scope are always verifiable and always count**: a\n changed row that shouldn't change, an email sent without approval, a post in\n the wrong channel. Every agent gets these checks regardless of Q1.\n- **Grade outcomes, not paths.** Two different sequences of tool calls that\n reach the right state both pass. Grade the path only where the path *is*\n the requirement (an approval step, a confirmation before a destructive\n action).\n- **\"Verifiable\" is per category.** An agent can have a verifiable \"create\n record\" category and an unverifiable \"advise\" category.\n\n**What the research adds.**\n- **Never grade from the agent's own report.** In single-control τ²-bench\n domains, 45–48% of failures are the agent confidently claiming success\n while the state says otherwise. Checks diff the real end state (mailbox,\n calendar, repo, sheet), never the chat reply.\n- **Write checkers against the goal, not one golden path.** Rule-based\n graders that expect a fixed sequence under-report success\n (AgentRewardBench); state-diff graders don't.\n- **Screen cases for validity.** SWE-bench Verified kept 500 of 2,294 tasks\n after human screening; a checklist audit found task-validity flaws in 7 of\n 10 agent benchmarks. Every case gets a human read plus a sanity run (a\n reference solution passes, a do-nothing agent fails).\n- **Keep some trajectory judging.** Outcome-only judges catch 84% of visible\n faults but 45% of silent ones (an action taken that the goal didn't ask\n for). A step-level judge on a sample, at ~3× cost, catches those.\n\n### Q2. Are you happy with performance? If so, optimize cost or latency?\n\nIf quality is acceptable, the eval's job changes: it becomes a **guardrail**\nthat lets you cut cost or latency without losing what works.\n\n| Situation | Eval shape |\n|---|---|\n| **Not happy with quality** | Quality eval first (Q1/Q6). Cost and latency are secondary metrics, recorded but not optimized |\n| **Happy; cost matters** | A non-regression suite (pass rate must not drop below the current baseline, per category) plus cost per successful task. Then experiments: cheaper model, shorter prompt, skip work when the input is unchanged, cache |\n| **Happy; latency matters** | Same suite plus latency per step and per turn (p50/p90), with a budget per category. Experiments: fewer tool calls, parallel steps, early exit |\n\nWhich one, cost or latency, depends on who waits:\n- **Scheduled runs** (crons): nobody waits. Optimize cost. Latency only\n matters against the wall-clock limit.\n- **Person-triggered runs**: someone is waiting for the reply. Optimize latency\n first; cost second.\n\nThe baseline must be measured before any change, with enough runs per case to\nsee variance (pass^k).\n\n**What the research adds.**\n- **Freeze a regression suite that passes ~100% today** and re-run it on every\n prompt, skill, tool or model edit; that is the floor cost work may not\n break.\n- **Report cost per successful task**, not cost per run, plus per-step\n latency percentiles.\n- **Prompt caching is the largest free lever** (cache reads at 0.1× input\n price), but any change to the prompt prefix (system prompt, tool\n definitions, a timestamp) invalidates it. Track cache-hit rate as a\n regression metric.\n- **Routing beats ensembles:** sending easy cases to a cheaper model gave\n >2× savings (RouteLLM); majority-vote and self-refine \"rarely justify the\n costs\".\n- **pass^k is unforgiving:** 75% per trial is ~42% pass^3. Decide which\n categories need every run to pass.\n\n### Q3. If performance is fine, what does drift look like, and how is it measured?\n\nDrift is the agent getting worse without anyone changing it on purpose. Causes:\nthe model behind the API changes; a skill or stored prompt is edited (by a\nperson or by the agent itself); a tool or API changes; the inputs change\n(new kinds of email, a new repo layout); user behaviour changes.\n\nQuantify it with two instruments:\n\n1. **A golden set re-run on a schedule.** A fixed set of cases (from Q1/Q6),\n run weekly and after every prompt, skill, model or harness change. Report\n pass rate per category and pass^k. A drop beyond normal variance is drift.\n2. **Online fingerprints from production**, per agent, per week, on a control\n chart: rate of silent turns on addressed asks, time to first reply, turns\n killed at the limit, tokens and cost per turn, tool calls per turn, the\n mix of categories, and any agent-specific outcome rate (approvals acted\n on, records saved, citations valid). A shift outside the usual\n band flags drift before anyone complains.\n\n\"Normal\" variance has to be measured first: run the golden set several times\non the same build to see the spread before reading any change as drift.\n\n**What the research adds.**\n- **Run-to-run noise is large:** temperature-0 accuracy varies by up to 15\n points across runs; 1,000 identical completions produced 80 distinct\n outputs. Only a change outside a k-run confidence interval is drift\n (Anthropic's error-bars guidance).\n- **Provider drift is real but bounded:** Anthropic pins weights per model\n ID; serving changes can still shift minor behaviour. The golden set\n catches it; so does logging the model ID per turn.\n- **Offline evals and A/B tests can both \"look good\" while the product gets\n worse** (OpenAI's 2025 sycophancy incident, driven by optimizing\n thumbs-up). Keep a weekly human read of 10–20 outlier traces; treat\n feedback buttons as monitoring inputs, never as the objective.\n\n### Q4. Can a sandbox be built?\n\nA sandbox is a copy of the agent's world where nothing real happens and the\nstarting state can be set. It is possible when the world is:\n\n| World | Sandbox | How |\n|---|---|---|\n| A git repo | Yes, trivially | A repo at a commit; grade the diff |\n| A structured SaaS API with few endpoints (Calendar, Sheets, a mail subset, a read-only finance API) | Yes | Stateful fakes seeded with recorded responses, behind a proxy |\n| A read-only API (GitHub, a finance API's GET endpoints) | Yes | Record and replay real responses |\n| The open web | No, but replayable | Record every page a real run fetched; replay; plant items for ground truth |\n| A rich editor (Slides, Docs) | Costly | Use a real scratch account instead |\n| A person in the loop | Partly | A scripted or simulated user for follow-ups |\n\nA sandbox also needs: a frozen clock, a fresh state per case, the agent's own\nfiles (anything that lives only in its sandbox), and the same model and\nharness as production.\n\n**What the research adds.**\n- **Fake only what changes state.** Record-and-replay for read-only tools\n (search, free/busy, GitHub, read-only finance endpoints); thin stateful fakes only for\n send/create/commit; scheduled contract tests of each fake against the real\n API so fakes don't drift.\n- **Isolate every trial.** Shared state between trials let an agent cheat by\n reading another trial's git history.\n- **The simulated user is the weakest part.** Unconstrained LLM users erred\n 40–47% of the time (12–13% critical) in τ²-bench; a tool-constrained one\n 16% (6%). Swapping the user model moved agent scores by up to 9 points.\n Script the user where possible; when simulating, constrain it with tools\n and ground it in real transcripts, and report scores\n per simulator.\n- **Validity compounds:** task × simulator × judge each at 70% ≈ 34% overall.\n\n### Q5. If no sandbox is possible, what then?\n\nThe eval moves from *before* deployment to *around* it:\n\n| Method | What it gives |\n|---|---|\n| **Recorded replay** | Re-run the agent against the tool outputs a real run saw. Not a true sandbox (the agent can't take a different path), but catches regressions on the recorded path |\n| **Shadow runs** | Run a candidate build on live inputs alongside production; compare outputs; nothing from the candidate is delivered |\n| **Canary** | Roll the change to one channel or a fraction of runs; watch the online fingerprints |\n| **Online judging** | A narrow judge scores a sample of live outputs on fixed criteria; trend it |\n| **Human spot checks** | A statistically sized sample reviewed each week in a trace review tool |\n| **Feedback capture** | Reactions, corrections, re-asks and edits after output, as labels |\n\nWithout a sandbox, \"eval\" mostly means **monitoring with a fixed instrument**,\nand every change must be reversible.\n\n**What the research adds.**\n- **Shadow mode before write access:** the agent states what it would do; a\n judge compares it with what the human did. Ramp staged its agent this way\n (99% \"improved\", 25% rejection rate) before letting it act.\n- **Online judge sampling rates:** 1–10% of traffic at high volume, 50–100%\n at low volume (Braintrust). For a low-volume agent, judge everything.\n- **The best production labels are behavioural:** edits made after the\n output (edit distance), acceptance rate, follow-up corrections and\n re-asks. Capture them; they cost nothing.\n\n### Q6. Are outputs too subjective to judge? If so, what criteria can be specified instead?\n\n\"Quality 1–10\" from a judge is not an eval. Replace it with **specified\ncriteria**: statements that are true or false of an output, each checkable\nby code or answerable by a narrow judge.\n\n| Approach | When |\n|---|---|\n| **Criteria checklist** | Most cases. \"Every claim has a source\", \"no personal details\", \"under N words\", \"mentions the two options\". Each item is pass/fail; the score is the count |\n| **Reference-guided judging** | When a gold answer exists (a human-written note, a known list of action items): grade against it with precision and recall |\n| **Pairwise comparison** | When only \"better or worse\" is definable: compare candidate vs baseline output, swapping order to cancel position bias |\n| **Human labels as the standard** | Always calibrate the judge: measure agreement with human labels on the same items before trusting it, and re-check after any judge change |\n| **Monitor only** | When no criteria can be written that the owner would sign off on. Then the output is not evaluated; it is watched (Q5), and the owner's reactions are the signal |\n\nThe test for \"too subjective\": can the agent's owner write down what a good\noutput must contain and must not contain? If yes, it's criteria. If no, it's\nmonitor-only until they can.\n\n**What the research adds.**\n- **One binary judge per failure mode**, each with its own labelled set:\n plan 100–200 human labels per failure mode, and report true-positive and\n true-negative rates, not raw agreement (Hamel & Shankar). Fewer than ~60\n labels gives intervals too wide to act on.\n- **Binary beats Likert:** \"the difference between 3 and 4 is subjective\".\n A 1–10 outcome-quality score is a triage signal for humans, not a judge\n target.\n- **Judge with a different model family** than the agent: self-preference\n correlates with self-recognition. Good judges reach ~80% agreement with\n humans, which is also the human–human ceiling.\n- **Pairwise needs position swapping** (position bias 50–70% without it).\n Use pairwise plus edit distance to *trend* subjective outputs (drafts,\n summaries, syntheses) rather than score them absolutely.\n- **Rubrics drift as you grade** (EvalGen): version the criteria, and\n re-calibrate after any change.\n- 82% of 55 agent-eval papers reported broken inter-rater statistics:\n compute and publish judge–human reliability per agent.\n\n---\n\n## Decision procedure\n\nFor each agent, per category:\n\n```\nQ1 verifiable?\n fully → state-diff checks; sandbox if Q4 allows, else recorded replay\n partly → code checks for the definite part + criteria (Q6) for the rest\n not → Q6: criteria checklist / reference-guided / pairwise; else monitor-only\n\nQ4 sandbox possible?\n yes → build it; cases = translations of reviewed traces + variants\n no → Q5: recorded replay + shadow/canary + online judging + spot checks\n\nQ2 happy with quality?\n no → quality eval is the priority; record cost and latency\n yes → non-regression suite as a guardrail; optimize cost (scheduled) or latency (person-triggered)\n\nQ3 for every agent, regardless:\n golden set re-run weekly and on every change; online fingerprints on a control chart\n```\n\nTwo cross-cutting rules, plus three the research insists on:\n- **Side-effect checks for everyone** (nothing changed outside scope, nothing\n sent without approval).\n- **Conversation-behaviour checks for every chat agent** (addressed asks get a\n reply; people talking to each other don't; replies in the agent's own\n threads get a turn).\n- **A frozen regression suite re-run on every edit**, with k runs and error\n bars; single-run scores mean little.\n- **State-diff grading only**; never grade from what the agent said it did.\n- **Every judge is binary, per failure mode, calibrated on 100–200 human\n labels, and from a different model family than the agent.**"},{"slug":"phase-1-understand-the-system","title":"Steps 1–4: Understand the system","markdown":"Steps 1–4. Goal: know what agents exist, where they run, how much they run, and what already measures them.\n\n## 1. Inventory the agents\n\nFor each agent: where it lives (surface/channel), what triggers it (human,\nschedule, other agents), its instructions (prompts, skills, stored scheduled\nprompts), its tools and credentials, the harness that runs it, and the model\nit actually uses.\n\n- Sources: deployment config, prompt/skill files, the harness's source,\n runtime config stored in its database, traces.\n- Watch for: agents with no prompt file (instructions stored only at\n runtime), per-agent model or provider overrides, side calls on different\n models (routing, \"should I respond\", titles, compaction), permission mode.\n- **Version history and stable window.** For each agent, list every change\n that could alter its behavior: prompt or skill edits, stored scheduled-prompt\n edits, model or provider changes, harness upgrades (which hit every agent at\n once). The **stable window** is the time since the last significant change.\n Ask the owner: \"How long has this agent been stable in production?\" and\n check the answer against the history. Version control is a good first\n source; changes made at runtime (stored prompts, model overrides) need the\n runtime's own records.\n- Output: an agent catalog, with a version history and stable window for each\n agent.\n\n## 2. Map the infrastructure and trace pipeline\n\nWhere each agent runs (provider, region, services, sandbox vs host), the\nhardware, the model providers it reaches, network egress controls. Then the\ntraces: how they are collected, which provider stores them, exact locations,\nwhat they contain (prompts, completions, tool I/O, tokens, cost), retention,\nand whether records in different sinks can be joined.\n\n- Fix labeling gaps before collecting data for evals: every trace must be\n attributable to an agent.\n- Output: an infrastructure doc with a data-flow diagram.\n\n## 3. Measure volume per agent\n\nRuns, model calls, tokens and cost per agent per day, split by trigger\n(scheduled vs human). Make it a script that regenerates the report.\n\n- Tells you how to sample (a 50-runs/day scheduled agent and a 2-runs/day\n human-driven agent need different case sets) and where cost concentrates.\n- Output: a volume report plus the script.\n\n## 4. Catalog existing evals and monitors\n\nTests, eval suites, judges, benchmarks, production monitors, user-feedback\ncapture — across the agent repos and neighboring projects. Note which are\nlive, which run automatically, which cover agent *quality* vs platform\ncorrectness.\n\n- Output: an evals inventory and the gap list (agents with no quality\n signal)."},{"slug":"phase-2-understand-the-usage","title":"Steps 5–7: Understand the usage","markdown":"Steps 5–7. Goal: know what each agent is asked to do, where it goes wrong, and have its raw traces in hand.\n\n## 5. Characterize usage from cheap sources\n\nPer agent: trigger mix, a task taxonomy with real examples, what it produces\nor changes, observed failure modes with counts, what success means, and what\ncould be checked by code vs judgement. Also cross-agent patterns: shared\nfailure modes, how people phrase requests, hand-offs between agents.\n\n- Sources: scheduled-prompt exports, telemetry snapshots, incident ledgers,\n the agents' own commits to their records.\n- Caveat to record: these sources are usually failure-biased (they keep\n failed runs, not successful ones).\n- Output: a usage-patterns doc, plus the list of questions only raw\n transcripts can answer.\n\n## 6. Survey similar public benchmarks\n\nFor each agent's job, the closest 2–4 public benchmarks: what they measure,\ntask format, grader type (state check, LLM judge, human), size, license.\nDecide what is reusable (datasets, graders, task formats) vs idea-only.\n\n- Verify every citation against a live paper or repo.\n- Output: a benchmarks doc with \"what to borrow\".\n\n## 7. Pull raw traces\n\nExport runs, transcript entries, tool calls, per-step model usage and\nscheduled-job definitions. The window must cover each agent's stable window\n(step 1), and reach back far enough to give at least 10 runs in each category. Read-only\ntransaction; output to an ignored data directory.\n\n- **Whoever has production access may have to run it.** Have the script ready\n before asking.\n- **Read-only transactions refuse temporary objects** (views, tables): put\n filters inline.\n- **Set the per-entry size limit generously** (100,000 characters, not\n 20,000). Tool outputs such as search results are large, and a cut-off entry\n loses exactly the data reviewers need.\n- **Write down the export's real structure** (formats, join keys, counts) and\n check every assumption the category drafts made against it before using them.\n- Output: a local, ignored dataset, the re-runnable pull script, and a notes\n file on its structure."},{"slug":"phase-3-build-the-evals","title":"Steps 8–12: Build the evals","markdown":"Steps 8–12. Goal: a calibrated eval suite per agent, organized by category.\n\n## 8. Define trace categories\n\nSort each agent's traces into **categories**: the distinct kinds of job the\nagent does. Each category gets a **worth**: how much it matters when that kind\nof trace goes wrong. Evals are organized by category: sampling, checks, cases\nand scores are all per category.\n\nExample, an email scheduling agent:\n\n| Category | What happened | Worth | Simple? |\n|---|---|---|---|\n| Nothing to do | No new email needs handling | Low: the only failure is acting when it shouldn't | Yes |\n| Suggest times | A thread needs meeting times proposed | Medium: wrong or clashing times cost a round trip | No |\n| Confirm and book | A time was agreed; confirm it and create the invite | High: a wrong or missing invite costs the meeting | No |\n\n- **Heuristics first.** Pick out the simple categories with code rules on\n the trace before any model is involved: a scheduled check where no new\n message or email arrived, a sweep that found no new items. Base each rule\n on the input (was there anything new?), not only on what the agent did, so\n a run that missed a new email isn't filed as \"Nothing to do\".\n- **Flag simple categories.** A simple (degenerate) category is one where the\n right behavior is trivial and a heuristic can check it on every run.\n Downstream it gets 3 spot-check examples instead of 10 (step 9), and its\n check runs over every trace in the category (step 10).\n- **Model for the rest.** Have a model group the remaining traces by what the\n agent was asked to do and what it did, using the usage patterns from\n step 5.\n- The agent's owner confirms, merges, renames, sets the simple flag and\n assigns worth.\n- Every trace gets exactly one category. Keep a small \"Other\" category and\n revisit it after review.\n- **Check the drafts on the traces.** A code-only script labels every run with\n the drafted rules and counts each category against its target. Fix drafts\n where rules misfire or a category can't reach its target.\n- **Cross-check every simple-category rule against an independent labeler.**\n A \"nothing new\" rule must exclude runs where the input couldn't be read: a\n broken connection also reports zero new items, and that is the failure the\n category would otherwise hide.\n- **Watch for filters in front of the agent.** If a gate (a \"should I respond\"\n check) drops messages before the agent runs, categories such as \"not\n addressed\" collapse. Count gate-only runs separately and look at what the\n gate dropped: silent failures hide there.\n- **Cost matrix.** For each category, price each kind of failure: what an\n incorrect result, a claim that wasn't carried out, no response, a partial\n result, and so on cost the user (None/Low/Medium/High). Worth is the headline;\n the matrix says where within a category the damage is (an \"Incorrect\" booking\n costs more than a slow one). Reviewers then don't judge impact trace by trace:\n it is set from the matrix, and they override only unusual cases. Overrides\n show where the matrix is wrong.\n- Output: a category list per agent, with a definition, an example trace, a\n worth, a simple flag and a cost-matrix row for each, plus the heuristic behind\n each simple category.\n\n## 9. Review at least 10 traces per category\n\nA person reviews at least 10 traces from each category and records, for each:\nits category, the errors in it, and the quality of the outcome. The review is\nabout judging outcomes; finding the patterns across traces is step 10.\n\n- **Sampling: categories set the number.** At least 10 traces per category,\n 3 per simple category, picked at random within the category from the\n agent's stable window (step 1). A category that is 1% of runs gets the same\n 10 as one that is 90%: the rare categories are usually where the worth is.\n An agent with three categories, one of them simple, gets 23 traces.\n- **Rare categories.** Fewer than 10 runs in the stable window: extend back\n across earlier versions and record each trace's version. Still fewer: take\n all of them and note it.\n- First question for every conversational agent: was the message addressed\n to the agent at all?\n- Tool: a trace review tool. AI drafts\n every field; the reviewer confirms or changes each one. It must show the\n data the agent worked from, not just its API calls, or outcomes can't be\n judged.\n- Review surfaces production bugs. Report each to the agent's owner as it is\n found, separately from the eval work. When one trace looks wrong, check the\n whole pattern in the pulled data (every similar run, what happened next)\n before proposing a fix: one odd trace is often a pattern, and the fix\n may belong in the platform rather than the agent.\n- Output: at least 10 reviewed traces per category (3 per simple category),\n each with category, errors and an outcome-quality score.\n\n## 10. Analyze errors, define checks, build cases\n\nTurn the reviews into evals. This step is programmatic, with the owner\nconfirming the result.\n\n- **Error analysis.** A model clusters the reviewers' error notes and tags\n into a failure taxonomy per category, with counts, weighted by worth.\n- **Checks.** Each failure mode becomes one binary check. Prefer code checks\n on end state (diffs, rows, events, posts, side effects outside scope). Keep\n LLM-judge checks narrow: one pass/fail question each, reasoning before\n verdict. Separate cross-agent checks (responded when addressed, finished\n within the time budget, claimed actions match actual changes, approvals\n acted on) from agent- and category-specific ones.\n- **Simple categories.** Their heuristic becomes a code check run on every\n trace in the category, not only the sampled ones.\n- **Cases.** The reviewed traces become the agent's cases, with the\n confirmed review as the expected outcome. Freeze inputs (message, thread,\n tool/state snapshot) and scrub secrets and personal data before committing.\n- Output: failure taxonomy, check spec and versioned case files, per agent\n and category.\n\n## 11. Calibrate judges\n\nUse the reviewed traces (step 9) as the human labels. Measure agreement (true-positive and\ntrue-negative rates, not just accuracy). Iterate the judge prompt until it\nagrees; re-check after any judge change. Use a different model from the\nagent under test where it agrees as well.\n\n- Output: judge prompts with measured agreement.\n\n## 12. Run the baseline\n\nGrade each case 3–5 times. Report pass rate and pass^k (all k runs pass) per\ncheck and per agent.\n\n- Output: the baseline scorecard."},{"slug":"principles","title":"Principles","markdown":"1. **Evals come from the agent's real traffic and real failures, not from\n generic benchmarks.** Public benchmarks supply grading designs; your\n traces supply the test cases.\n2. **Understand before measuring.** Inventory, infrastructure, volume and\n usage come first. You can't pick what to check until you know what the\n agent is asked to do and where it goes wrong.\n3. **Grade outcomes, with code first.** Check what the agent changed in the\n world (records, rows, events, posts). Use an LLM judge only for what code\n can't decide, and calibrate it against human labels.\n4. **Cheap sources first, raw transcripts second.** Metadata, logs, commit\n history and incident ledgers characterize most of the picture without\n touching sensitive content. Pull full transcripts once you know what to\n look for.\n5. **Keep sensitive data out of version control.** Raw traces stay local and\n ignored; only reviewed, scrubbed cases get committed.\n6. **Record confirmed vs inferred** for every claim, with a source. Config\n often disagrees with runtime; trust what the traces show."}],"practices":{"title":"Eval practice: findings from the 2025–2026 literature","intro":"What the literature says about evaluating production LLM agents. Single-author 2026 preprints\nare marked as such in the source list: directional, not settled.\n\nCitations are short tags like [Anthropic 2026]; each resolves to an entry under\n[Sources](#sources).","sections":[{"id":"1-verifiability","title":"1. Verifiability","markdown":"**Finding.** Grade the end state with code wherever the task leaves one, and\ngrade it against the *goal*, not one expected path: rule-based checkers that\nexpect a fixed sequence under-report success, and agents find valid paths the\nauthor didn't anticipate [Anthropic 2026, AgentRewardBench]. Never take the\nagent's own report as the signal: 45–48% of failures in comparable\nsingle-control benchmarks were the agent confidently claiming success while\nthe state said otherwise [False success]. Keep trajectory judging for hard\nconstraints (a required approval before a destructive step) and for silent\nfaults: outcome-only judges caught 84% of user-visible faults but only 45% of\nsilent ones [trajectory-judge]. \"Verifiable\" is a property you establish per\ntask by human screening; SWE-bench kept 500 of 2,294 tasks after developers\nchecked them, and fixing grader bugs alone moved one Anthropic score from 42%\nto 95% [SWE-bench Verified, Anthropic 2026]."},{"id":"2-optimizing-cost-or-latency-once-quality-is-acceptable","title":"2. Optimizing cost or latency once quality is acceptable","markdown":"**Finding.** When quality is acceptable, the eval becomes a guardrail: freeze\na regression suite the agent passes today at ~100%, measure cost per\n*successful* task (not per run), per-step latency percentiles and cache-hit\nrate, then run cheaper configurations against that floor with paired\ncomparisons [Anthropic 2026, Cost-of-Pass, Error bars]. Inference-time tricks\n(majority vote, self-refine) \"rarely justify the costs\" without a real\nverifier [Cost-of-Pass, Monkeys]; routing easy cases to a cheaper model cut\ncost by more than 2× in some cases without quality loss [RouteLLM]; ablating\nmodules (planning depth, tool set, memory) kept 96.7% of performance at 43%\nless cost in one study [Efficient Agents]. Prompt caching is the largest free\nlever (cache reads at 0.1× input price, \"up to 90%\" cheaper on long prompts)\nbut any edit to the prompt prefix, tool definitions or a timestamp in it\ninvalidates the cache [Caching]. pass^k is unforgiving: 75% per trial is\nabout 42% pass^3 [Anthropic 2026]."},{"id":"3-drift","title":"3. Drift","markdown":"**Finding.** \"Drift\" bundles five different things: the provider's model\nchanging, the team's own prompt and skill edits, upstream API changes, a shift in the\ninputs, and run-to-run nondeterminism that only looks like drift. Practice\nseparates them: pin model IDs so provider changes are observable (Anthropic\ndoesn't change weights behind an ID, but serving changes can shift minor\nbehaviour) [Model IDs]; make every prompt or skill edit a CI event that\nre-runs the regression suite; contract-test the fakes against the real APIs;\ntest the inputs for distribution shift [Evidently]; and measure normal\nvariance first, because temperature-0 accuracy varies by up to 15 points\nbetween runs and 1,000 identical completions produced 80 distinct outputs\n[Nondeterminism, Thinking Machines]. Same-name models have moved by tens of\npoints in a quarter [Chen 2023]. The canonical warning is OpenAI's sycophancy\nregression: offline evals and A/B tests both looked good, because nobody had\na metric for the thing that changed and thumbs-up was the optimization\ntarget; only expert readers noticed [Sycophancy]. No source gives a universal\n\"normal drift\" number; the honest answer is your own k-run confidence\ninterval [Error bars]."},{"id":"4-sandboxes","title":"4. Sandboxes","markdown":"**Finding.** Four designs recur: hand-written fakes (full control, high\nmaintenance), record-and-replay cassettes (exact for the recorded path,\nuseless off it), staging accounts (high fidelity, slow, flaky), and\nself-hosted real software with seeded data [TheAgentCompany, Agent VCR].\nThe consistent rules: isolate and reset state per trial (leftover git history\nlet an agent cheat across trials), freeze time and seeds, grade the end\nstate, and treat the simulated user as the least reliable part [Anthropic\n2026]. Numbers on that last point: unconstrained LLM users erred 40–47% of\nthe time (12–13% critically) versus 16% (6%) when constrained by tools\n[τ²-bench]; agent scores moved up to 9 points just by changing the\nsimulator's model [Lost in Simulation]; personas written by hand matched\nreal users on only 6–8% of stylistic dimensions, and grounding simulators in\nreal conversations raised that to 45% and exposed hidden failures\n[RealUserSim]. Validity compounds: task × simulator × judge each at 70% is\nabout 34% overall [Validity]. Before trusting an environment, a trivial agent\nshould score ~0 and a reference solution ~100 [ABC checklist]."},{"id":"5-when-the-world-cant-be-faked","title":"5. When the world can't be faked","markdown":"**Finding.** For open-web research, live SaaS and human-in-the-loop flows,\nthe eval moves into production instead of pretending: shadow mode (the agent\nsays what it would do; a judge compares it with what the human did) before\nwrite access [Ramp]; canary cohorts; online reference-free judges on a sample\nof live traffic (1–10% at high volume, 50–100% at low) [Braintrust, LangSmith,\nDatadog evals]; a fixed weekly human quota [Hamel FAQ]; and the user's own\nbehaviour as the label: edits made after the output, acceptance, follow-up\ncorrections, which the literature finds more honest than thumbs [PRELUDE,\nCopilot]. Ramp staged its merchant-classification agent this way (manual\nreview, then follow-up rate, then a 25% rejection rate, then a judge finding\nabout 99% of proposals improved) before letting it act [Ramp]. Cheap\nfalse-success detectors work as the first monitor, with judges only on the\nflagged subset [False success]."},{"id":"6-subjective-outputs","title":"6. Subjective outputs","markdown":"**Finding.** The consensus procedure: replace \"rate quality 1–10\" with\nbinary pass/fail on named criteria, one judge per failure mode (\"the\ndifference between 3 and 4 is subjective\") [Hamel judge, Hamel FAQ]; build\neach judge from one domain expert's labelled critiques and report\ntrue-positive and true-negative rates, not raw agreement (a judge that always\nsays pass still \"agrees\" 95% of the time); plan 100–200 labelled examples per\nfailure mode, since below about 60 the intervals are too wide [Hamel judge,\nHamel FAQ]. Use pairwise comparison with position swapping for \"better or\nworse than before\" (position bias is 50–70% without the swap) and a\ndifferent model family for the judge, because self-preference tracks\nself-recognition [MT-Bench, Yan, Self-preference]. Expect the criteria to\nchange while you grade, and version them [Validators]. Good judges reach\nabout 80% agreement with humans, which is also the human–human ceiling\n[MT-Bench]. Checklists help most when items are objective and the mode is\npairwise [TICK, Checklists]. 82% of 55 agent-eval papers reported broken or\nmissing inter-rater statistics; publish yours [Validity]. And some outputs\ncan only be monitored, not scored: when two experts can't agree pass/fail\nafter seeing each other's critiques, track pairwise trends and edit distance\ninstead [Hamel judge, Validators]."},{"id":"sources","title":"Sources","markdown":"Fetched and checked 2026-09-30. Grouped by the section that leans on them;\nseveral are cited in more than one.\n\n### Verifiability\n\n- **[Anthropic 2026]** Anthropic, \"Demystifying evals for AI agents\",\n 2026-01-09. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents.\n Grade what the agent produced, not the path; code, model and human graders,\n \"choose deterministic where possible\"; partial credit; capability vs\n regression evals; pass^k (75% per trial ≈ 42% pass^3); isolate every trial;\n CORE-Bench 42% → 95% after fixing grader bugs.\n- **[Multi-agent]** Anthropic, \"How we built our multi-agent research\n system\", 2025-06-13. https://www.anthropic.com/engineering/multi-agent-research-system.\n Judge the final state for state-changing agents; one rubric judge emitting\n 0–1 scores plus pass/fail was most aligned with humans; ~20 real queries\n were enough early on; humans still found what evals missed.\n- **[τ-bench]** Sierra, \"τ-bench\", 2024-06-17. https://arxiv.org/abs/2406.12045.\n Compares the database state at the end with the annotated goal state;\n introduced pass^k; gpt-4o <50% success and pass^8 <25% on retail.\n- **[AgentRewardBench]** 2025-04-11, rev. 2025-10. https://arxiv.org/abs/2504.08942.\n 1,302 expert-labelled trajectories; rule-based evaluation \"tends to\n underreport the success rate\"; no single LLM judge best everywhere.\n- **[ABC checklist]** \"Establishing Best Practices for Building Rigorous\n Agentic Benchmarks\", 2025-07-03. https://arxiv.org/abs/2507.02825. Task\n validity, outcome validity, reporting; flaws mis-estimate performance \"by\n up to 100% in relative terms\"; trivial-agent and reference-solution runs.\n- **[SWE-bench Verified]** OpenAI + SWE-bench, 2024-08.\n https://www.swebench.com/verified.html (OpenAI's post\n https://openai.com/index/introducing-swe-bench-verified/ blocks fetches).\n 500 tasks kept after developers confirmed each was clear, correct and\n solvable.\n- **[False success]** \"From Confident Closing to Silent Failure:\n Characterizing False Success in LLM Agents\", 2026-06-01, preprint.\n https://arxiv.org/abs/2606.09863. 45–48% of failures in single-control\n τ²-bench domains are false success claims; 3% in dual-control telecom;\n 75.8% among self-assessing coding agents; cheap TF-IDF detectors beat LLM\n judges for triage (AUROC 0.83–0.95).\n- **[trajectory-judge]** \"What Outcome-Only LLM Judges Miss\", 2026-08-29,\n preprint. https://arxiv.org/abs/2609.00038. Outcome judges: 84% of visible\n faults, 45% of silent ones, 33% false alarms; step judges: 77% of silent\n faults, zero false alarms, ~3× cost.\n- **[RLVR]** Tülu 3, AI2, 2024-11-22, https://arxiv.org/abs/2411.15124; and\n \"Let's Verify Step by Step\", OpenAI, 2023-05-31,\n https://arxiv.org/abs/2305.20050. Verification functions replace learned\n judges where outcomes are verifiable; process supervision pays where\n outcomes are cheap to fake.\n- **[ADK]** Google ADK evaluation docs. https://adk.dev/evaluate/. Ships\n strict trajectory matching by default (EXACT / IN_ORDER / ANY_ORDER): the\n opposite of Anthropic's advice; use only for true invariants.\n\n### Cost and latency\n\n- **[Cost-of-Pass]** Stanford, 2025-04-17. https://arxiv.org/abs/2504.13359.\n Expected dollars per correct solution; majority voting and self-refinement\n \"rarely justify the costs\"; cost halves every few months on hard tasks.\n- **[Efficient Agents]** 2025-07-24. https://arxiv.org/abs/2508.02694. Kept\n 96.7% of performance while cutting $0.398 → $0.228 per task by ablating\n modules.\n- **[RouteLLM]** LMSYS, 2024-06-26 (ICLR 2025). https://arxiv.org/abs/2406.18665.\n Preference-trained routing: \"cost reductions—by over 2 times in certain\n cases—without compromising the quality\".\n- **[Monkeys]** \"Large Language Monkeys\", 2024-07-31.\n https://arxiv.org/abs/2407.21787. Coverage scales with samples only where\n an automatic verifier exists.\n- **[Caching]** Anthropic prompt caching: pricing\n https://platform.claude.com/docs/en/about-claude/pricing; guide\n https://platform.claude.com/docs/en/build-with-claude/prompt-caching;\n launch post https://claude.com/blog/prompt-caching. Cache read 0.1× input\n (0.05× Opus 5.5, 0.025× Fable 5.1); \"up to 90%\" cost and \"up to 85%\"\n latency on long prompts; any prefix edit invalidates.\n- **[Pricing]** Anthropic pricing, fetched 2026-09-30. Opus 5.5 $4/$20 per\n MTok, Sonnet 5 $2/$10, Haiku 4.5 $1/$5, Fable 5.1 $10/$50; Batch −50%;\n the 4.7+ tokenizer yields ~30% more tokens for the same text.\n- **[Datadog metrics]** Datadog Agent Observability metrics.\n https://docs.datadoghq.com/llm_observability/monitoring/metrics/. Span\n duration as distributions by span kind and model; tokens, cost, errors,\n eval scores on spans.\n- **[Error bars]** Miller (Anthropic), \"Adding Error Bars to Evals\",\n 2024-11-01. https://arxiv.org/abs/2411.00640. Paired differences on the\n same task set, clustered standard errors, resampling, power analysis.\n\n### Drift\n\n- **[Chen 2023]** Chen, Zaharia, Zou, \"How is ChatGPT's behavior changing\n over time?\", 2023-07-18. https://arxiv.org/abs/2307.09009. GPT-4 prime\n identification 84% → 51% in three months.\n- **[Model IDs]** Anthropic, \"Model IDs and versioning\", fetched 2026-09-30.\n https://platform.claude.com/docs/en/about-claude/models/model-ids-and-versions.\n Weights behind an ID don't change; serving infrastructure can, with\n \"minor differences in observable behavior\".\n- **[Sycophancy]** OpenAI, \"Expanding on what we missed with sycophancy\",\n 2025-05-02. https://openai.com/index/expanding-on-sycophancy/ (blocks\n fetches; quotes verified at\n https://simonwillison.net/2025/May/2/what-we-missed-with-sycophancy/).\n Offline evals and A/B tests looked good; expert testers said it \"felt\"\n off; thumbs-up as a reward signal was the cause.\n- **[Nondeterminism]** \"Non-Determinism of 'Deterministic' LLM Settings\",\n 2024-08-06, rev. 2025-04. https://arxiv.org/abs/2408.04667. Accuracy varied\n up to 15 points across temperature-0 runs.\n- **[Thinking Machines]** \"Defeating Nondeterminism in LLM Inference\",\n 2025-09-10. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/.\n 1,000 identical completions, 80 distinct outputs; batch-size-dependent\n kernels.\n- **[Evidently]** \"5 methods to detect drift in ML embeddings\", 2023-05-17,\n updated 2025-07-16. https://www.evidentlyai.com/blog/embedding-drift-detection.\n Domain-classifier method with ROC-AUC ≈ 0.55 threshold as the default for\n text.\n- **[Hamel FAQ]** Hamel Husain & Shreya Shankar, evals FAQ, 2025, updated\n 2026-09. https://hamel.dev/blog/posts/evals-faq/. Re-run error analysis on\n significant changes; review 10–20 outlier traces weekly; binary over\n Likert; 100–200 labels per failure mode; \"60–80% of our development time\n on error analysis and evaluation\".\n\n### Sandboxes\n\n- **[τ²-bench]** Sierra, 2025-06-09. https://arxiv.org/abs/2506.07982.\n User-simulator error rates: retail 40% (12% critical), airline 47% (13%),\n tool-constrained telecom 16% (6%); dual control drops agent pass^1 by\n 18–25 points.\n- **[Lost in Simulation]** 2026-01-23. https://arxiv.org/abs/2601.17087.\n Success varies up to 9 points across user LLMs; simulators misestimate by\n task difficulty and dialect.\n- **[RealUserSim]** Salesforce, 2026-04-07. https://arxiv.org/abs/2605.20204.\n Hand-written personas match real users on 6–8% of stylistic dimensions;\n grounding in 14K real conversations raised it to 45.3% and exposed hidden\n failures.\n- **[TheAgentCompany]** CMU, 2024-12-18, rev. 2025-09.\n https://arxiv.org/abs/2412.14161. Self-hosted GitLab, RocketChat,\n ownCloud, Plane with seeded data; checkpoint-based partial credit; best\n agent ~30% of 175 tasks.\n- **[Agent VCR]** Capital One, 2025. https://github.com/Jarvis2021/agent-vcr.\n Records MCP traffic to cassettes; replays as a mock server; diffs\n recordings across versions; five match modes.\n\n### No sandbox\n\n- **[Ramp]** \"Fixing merchant classifications with AI\", 2025.\n https://builders.ramp.com/post/fixing-merchant-classifications-with-ai.\n Shadow mode; staged signals; 25% rejection rate; judge found ~99%\n improved; live actions only after shadow results held.\n- **[Braintrust]** Online scoring docs. https://www.braintrust.dev/docs/platform/logs/score.\n Sample 1–10% at high volume, 50–100% at low; promote traces into the\n offline set.\n- **[LangSmith]** Evaluation concepts. https://docs.langchain.com/langsmith/evaluation-concepts.\n Offline vs online; pairwise when direct scoring is hard; backtesting on\n historical inputs.\n- **[Datadog evals]** https://docs.datadoghq.com/llm_observability/evaluations/.\n Custom judge prompts over traces; external results and end-user feedback\n attached to the same spans.\n- **[PRELUDE]** PRELUDE / CIPHER, NeurIPS 2024, 2024-04-23.\n https://arxiv.org/abs/2404.15269. The user's edits as the feedback signal;\n cost = edit distance.\n- **[Copilot]** GitHub Copilot productivity study, 2022-05-13.\n https://arxiv.org/abs/2205.06537. Acceptance rate tracks perceived\n productivity better than persistence.\n\n### Subjective outputs\n\n- **[Hamel judge]** Hamel Husain, \"Creating a LLM-as-a-Judge That Drives\n Business Results\", 2024-10-29. https://hamel.dev/blog/posts/llm-judge/.\n Critique shadowing; ~100 labels per failure mode; below 60 the intervals\n are too wide; report TPR and TNR; Honeycomb converged in three iterations.\n- **[Validators]** Shankar et al., \"Who Validates the Validators?\", UIST\n 2024. https://arxiv.org/abs/2404.12272. Criteria drift: grading outputs\n changes the criteria.\n- **[MT-Bench]** Zheng et al., \"Judging LLM-as-a-judge\", NeurIPS 2023.\n https://arxiv.org/abs/2306.05685. Position, verbosity and self-enhancement\n bias; >80% agreement with humans, the human–human level.\n- **[Yan]** Eugene Yan, \"Evaluating the Effectiveness of LLM-Evaluators\",\n 2024-08. https://eugeneyan.com/writing/llm-evaluators/. Direct scoring for\n objective checks, pairwise for subjective; position bias 50–70%; a few\n hundred labels to align.\n- **[Self-preference]** Panickssery, Bowman, Feng, \"LLM Evaluators Recognize\n and Favor Their Own Generations\", 2024-04-15. https://arxiv.org/abs/2404.13076.\n- **[Arena]** Chatbot Arena, 2024-03-07. https://arxiv.org/abs/2403.04132.\n Pairwise votes → Bradley-Terry ranking with confidence intervals.\n- **[TICK]** 2024-10-04. https://arxiv.org/abs/2410.03608. YES/NO checklists\n raised LLM–human agreement 46.4% → 52.2%.\n- **[Checklists]** \"Are Checklists Really Useful?\", EMNLP 2025, 2025-08-21.\n https://arxiv.org/abs/2508.15218. Gains hold in pairwise settings, less in\n direct scoring.\n- **[Validity]** Caban, \"Measurement Without Validity\", 2026-08, preprint.\n https://arxiv.org/abs/2608.00794. Validity compounds across task ×\n simulator × judge; 82% of 55 papers report broken inter-rater statistics;\n proposes ICC ≥ 0.70."}]},"benchmarks":{"vocabulary":{"workloads":["code-review","coding","computer-use","crm","customer-support","data-analysis","devops","documents","education","email-calendar","embodied","finance","games","general-assistant","government","healthcare","hr","insurance","it-ops","knowledge-qa","legal","logistics","manufacturing","memory","ml-engineering","mobile","multi-agent","planning","real-estate","research","retail","safety","sales","science","security","shopping","spreadsheets","sql","telecom","tool-calling","travel","web-browsing","workflow-automation"],"interaction":["autonomous-long-horizon","multi-turn-scripted","multi-turn-simulated-user","single-turn"],"environment":["api-sandbox","browser","containers","database","desktop","filesystem","mixed","mobile","static","terminal"],"grading":["exact-match","human","llm-judge","pass^k","programmatic-check","state-diff","trajectory-check","unit-tests"],"harness":["agentbeats","hal","inspect_evals","none","own-repo"],"good_for":["baseline","case-template","grading-design","privacy-pattern","sandbox-design","taxonomy","user-simulator"],"status":["active","deprecated","frozen","saturated"],"data":["mixed","real-public","real-scraped","synthetic"],"pii":["none","possible","yes"]},"entries":[{"id":"contextcrbench","name":"ContextCRBench","year":2025,"by":"ByteDance and Harbin Institute of Technology (Hu et al.)","workloads":["code-review"],"task_types":["judge whether a diff hunk needs review comments","point to the exact lines with defects","write the review comment a human would leave, given the linked issue and surrounding code"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["programmatic-check","llm-judge"],"size":67910,"data":"real-scraped","pii":"possible","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2511.07017","github":"https://github.com/kinesiatricssxilm14/ContextCRBench","leaderboard":null},"good_for":["taxonomy","case-template"],"fit":"Fits teams building a review bot with issue context; borrow the three granularities (hunk, line, comment) as a taxonomy and the context-enrichment fields per record.","caveats":"Scraped from 100 public repos with no stated license and reviewer names possibly included; metrics are not described in the README; very large so sample it."},{"id":"swe-prbench","name":"SWE-PRBench","year":2026,"by":"Deepak Kumar","workloads":["code-review"],"task_types":["find the issues a human reviewer flagged in a pull request diff","review with diff only, diff plus files, or full repository context"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge"],"size":350,"data":"real-public","pii":"possible","license":"CC-BY-4.0","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2603.26130","github":null,"leaderboard":null},"good_for":["grading-design","taxonomy"],"fit":"Fits code-review agents that choose how much context to load; its finding that more context lowered every model's score is directly useful for prompt design.","caveats":"Single author, LLM-judged (kappa 0.75), and the repository URL is not on the paper page; models found only 15 to 31 percent of issues, so scores cluster near the floor."},{"id":"swr-bench","name":"SWR-Bench","year":2025,"by":"Peking University and others","workloads":["code-review"],"task_types":["review a whole pull request with full project context and point out real issues","stay quiet on a clean pull request instead of inventing problems"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge"],"size":1000,"data":"real-public","pii":"possible","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2509.01494","github":null,"leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits pull-request review bots; borrow the structured ground-truth issue list and the coverage-based judge (about 90 percent agreement with humans) as a review rubric.","caveats":"Judge-based scoring, no public repository link on the paper page, and the paper title and dates differ across versions, so confirm which release you use."},{"id":"aider-polyglot","name":"Aider polyglot benchmark","year":2024,"by":"Aider (Paul Gauthier)","workloads":["coding"],"task_types":["solve an Exercism exercise in C++, Go, Java, JavaScript, Python or Rust by editing files","fix the solution after seeing test failures on a second attempt"],"interaction":"multi-turn-scripted","environment":"containers","state_checked":true,"tools":["file editor","test runner"],"grading":["unit-tests"],"size":225,"data":"real-public","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://aider.chat/2024/12/21/polyglot.html","github":"https://github.com/Aider-AI/polyglot-benchmark","leaderboard":"https://aider.chat/docs/leaderboards/"},"good_for":["baseline","grading-design"],"fit":"Fits chat-style code editing tools that apply diffs to files; borrow the two-attempt protocol (pass rate 1 vs 2) that measures recovery from test feedback.","caveats":"Top models are near 88 percent so headroom is small; exercises are public Exercism problems (likely memorized) and the content license belongs to Exercism."},{"id":"apps","name":"APPS","year":2021,"by":"UC Berkeley and others (Hendrycks et al.)","workloads":["coding"],"task_types":["turn a natural-language problem statement into a Python program","pass hidden test cases at introductory, interview and competition levels"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["unit-tests"],"size":10000,"data":"real-scraped","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2105.09938","github":"https://github.com/hendrycks/apps","leaderboard":null},"good_for":["baseline"],"fit":"Still used as a large, cheap code-generation baseline; its test-case checker is reused by LiveCodeBench, so it is a known-good grading script.","caveats":"From 2021 and fully in training data; some test cases are known to be wrong; not an agent benchmark."},{"id":"ask-or-assume","name":"Ask or Assume? (clarification-seeking in coding agents)","year":2026,"by":"Edwards and Schuster","workloads":["coding","general-assistant"],"task_types":["notice that a software issue description is underspecified","ask a clarifying question instead of guessing","resolve the issue once the missing detail is supplied"],"interaction":"multi-turn-simulated-user","environment":"containers","state_checked":true,"tools":["shell","code editing","test runner"],"grading":["unit-tests"],"size":null,"data":"real-public","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2603.26233","github":null,"leaderboard":null},"good_for":["case-template","user-simulator"],"situations":["ambiguous"],"fit":"Fits coding agents and any agent that should ask before acting on vague requests. Borrow the recipe of stripping details from SWE-bench Verified issues and scoring both the resolve rate and how many questions were asked.","caveats":"Not a named benchmark release; the paper evaluates its own scaffold on an underspecified SWE-bench variant, task count and code link are not on the paper page, paper is CC BY-NC-SA."},{"id":"bigcodebench","name":"BigCodeBench","year":2024,"by":"BigCode project (Zhuo et al.)","workloads":["coding","tool-calling"],"task_types":["write a Python function that combines calls to several libraries from a docstring","follow a short natural-language instruction to produce a tested function"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["unit-tests"],"size":1140,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["inspect_evals","own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2406.15877","github":"https://github.com/bigcode-project/bigcodebench","leaderboard":"https://huggingface.co/spaces/bigcode/bigcodebench-leaderboard"},"good_for":["baseline","grading-design"],"fit":"A function-level baseline for library-heavy Python; borrow its high-coverage test writing (about 5.6 tests per task, 99 percent branch coverage).","caveats":"The repository was archived in July 2026, so it will not be updated; single-function, no repo context; the 148-task Hard subset is the only part with headroom."},{"id":"bugsinpy","name":"BugsInPy","year":2020,"by":"Singapore Management University (Widyasari et al.)","workloads":["coding"],"task_types":["fix a real bug in one of 17 Python projects so its test passes","reproduce a bug in a pinned environment"],"interaction":"single-turn","environment":"containers","state_checked":true,"tools":["shell","pytest"],"grading":["unit-tests"],"size":493,"data":"real-public","pii":"none","license":"unknown","harness":["own-repo"],"status":"frozen","links":{"paper":"https://2020.esec-fse.org/details/esecfse-2020-tool-demos/26/BugsInPy-A-Database-of-Existing-Bugs-in-Python-Programs-to-Enable-Controlled-Testing","github":"https://github.com/soarsmu/BugsInPy","leaderboard":null},"good_for":["case-template"],"fit":"A Python counterpart to Defects4J for repair agents; useful as a small, reproducible pilot set before building your own.","caveats":"Old and memorized, environments are known to be hard to rebuild, and later papers quote 501 bugs instead of 493, so counts are inconsistent."},{"id":"classeval","name":"ClassEval","year":2023,"by":"Fudan University","workloads":["coding"],"task_types":["generate a whole Python class with several interdependent methods from a skeleton","keep methods consistent with each other when generated one at a time"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["unit-tests"],"size":100,"data":"synthetic","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2308.01861","github":"https://github.com/FudanSELab/ClassEval","leaderboard":null},"good_for":["grading-design"],"fit":"Fits code generators that emit whole modules; borrow class-level versus method-level pass@1 reporting to show where multi-part outputs break.","caveats":"Only 100 hand-written classes (410 methods); data is CC BY-NC 4.0 (non-commercial); 2023 vintage and likely memorized."},{"id":"commit0","name":"Commit0","year":2024,"by":"Cornell, Stanford, NYU (Zhao et al.)","workloads":["coding"],"task_types":["implement a whole Python library from its specification and a test suite","iterate on failing unit tests, lint and type checks until the library works"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor","pytest"],"grading":["unit-tests","programmatic-check"],"size":57,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2412.01769","github":"https://github.com/commit-0/commit0","leaderboard":"https://commit-0.github.io/"},"good_for":["grading-design","case-template"],"fit":"Fits greenfield coding agents that build from a spec rather than patch; borrow the \"percent of tests passed\" partial credit and the lint/type-check gates.","caveats":"Only 57 libraries, all well-known Python packages whose source is in training data, so memorization is a risk; runs are long and costly."},{"id":"cruxeval","name":"CRUXEval","year":2024,"by":"Meta AI, MIT (Gu et al.)","workloads":["coding"],"task_types":["predict the output of a short Python function for a given input","predict an input that produces a given output"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match"],"size":800,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2401.03065","github":"https://github.com/facebookresearch/cruxeval","leaderboard":"https://crux-eval.github.io/leaderboard.html"},"good_for":["grading-design"],"fit":"Fits teams checking whether a model actually reads code before editing it; borrow the execution-prediction task as a cheap probe inside a bigger code-review suite.","caveats":"Functions are 3 to 13 lines of LLM-generated Python; GPT-4 was already at 75 to 81 percent in 2024, so current models are near the ceiling."},{"id":"defects4j","name":"Defects4J","year":2014,"by":"University of Washington (Just, Jalali, Ernst)","workloads":["coding"],"task_types":["fix a real Java bug so the triggering JUnit test passes","localize the faulty method from failing tests"],"interaction":"single-turn","environment":"containers","state_checked":true,"tools":["shell","java build"],"grading":["unit-tests"],"size":854,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"frozen","links":{"paper":"https://homes.cs.washington.edu/~mernst/pubs/defects4j-issta2014.pdf","github":"https://github.com/rjust/defects4j","leaderboard":null},"good_for":["case-template","baseline"],"fit":"Still the reference set for Java program-repair agents (RepairAgent, ReinFix report on it); borrow the minimized single-commit fix and triggering-test rules for writing repair cases.","caveats":"From 2014, fully in training data, and results are split across v1.2 and v2.0 subsets that papers mix up; not an agent benchmark on its own."},{"id":"devbench","name":"DevBench (DevEval)","year":2024,"by":"ByteDance, Shanghai AI Lab (Li et al.)","workloads":["coding"],"task_types":["write a software design document from a product requirements document","set up the environment and implement a small repo in Python, C/C++, Java or JavaScript","write acceptance and unit tests for the implementation"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests","llm-judge"],"size":22,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2403.08604","github":"https://github.com/open-compass/DevBench","leaderboard":null},"good_for":["taxonomy","case-template"],"fit":"Fits agents covering more than coding, such as design docs and test writing; borrow its lifecycle stages as a taxonomy for an agent's golden set.","caveats":"Only 22 repos, so results are noisy; the design-document stage is LLM-judged; data is CC BY 4.0 while code is Apache-2.0."},{"id":"fea-bench","name":"FEA-Bench","year":2025,"by":"Peking University, Microsoft (Li et al.)","workloads":["coding"],"task_types":["add a feature to a repository given a description and the new function signatures","make new unit tests from the feature pull request pass"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests"],"size":null,"data":"real-public","pii":"possible","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2503.06680","github":null,"leaderboard":null},"good_for":["case-template"],"fit":"Fits incremental-development agents in Python; the task format (issue plus new component signatures plus tests) is a usable template for feature cases.","caveats":"Instance count and data license were not visible on the paper page; the code link is not on the arXiv page, so check availability before planning on it."},{"id":"frontier-cs","name":"FrontierCS","year":2025,"by":"Princeton and others (Cheng et al.)","workloads":["coding","research"],"task_types":["write a program that scores as well as possible on an open-ended algorithmic optimization problem","solve a research-style systems, HPC, database or security task with no known optimal answer"],"interaction":"single-turn","environment":"containers","state_checked":false,"tools":["none"],"grading":["programmatic-check"],"size":248,"data":"mixed","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2512.15699","github":"https://github.com/FrontierCS/Frontier-CS","leaderboard":"https://github.com/FrontierCS/Frontier-CS/tree/main/leaderboard"},"good_for":["grading-design"],"fit":"Fits agents that optimize rather than just pass; borrow the continuous scoring curve (60 points for matching the reference, log bonus for beating it, capped at 100).","caveats":"Size is 180 algorithmic plus 68 research problems and keeps growing, so pin a version; submissions go by email, so the leaderboard is not independently run."},{"id":"hcast","name":"HCAST","year":2025,"by":"METR","workloads":["coding","ml-engineering","security","general-assistant"],"task_types":["complete a software engineering task from a spec in a container","run a machine learning experiment end to end","solve a cybersecurity challenge","carry out a multi-hour reasoning task with human time baselines"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell in a Linux container"],"grading":["programmatic-check"],"size":189,"data":"mixed","pii":"none","license":"MIT","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2503.17354","github":null,"leaderboard":null},"good_for":["baseline","taxonomy","grading-design"],"fit":"Fits teams who want to express agent ability as \"how long a human would take\". Borrow the human-time calibration (563 baselines) and the algorithmic scorer per task.","caveats":"Most tasks are kept private by METR; software-heavy rather than office work; no public harness."},{"id":"humanevalfix","name":"HumanEvalFix (HumanEvalPack)","year":2023,"by":"BigCode project (Muennighoff et al.)","workloads":["coding"],"task_types":["fix a bug in a short function given failing tests, in six languages","explain a function and re-synthesize it from the explanation"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["unit-tests"],"size":984,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2308.07124","github":"https://github.com/bigcode-project/octopack","leaderboard":null},"good_for":["case-template"],"fit":"A tiny bug-fix template: 164 HumanEval problems times six languages with planted bugs; useful only as a quick multi-language smoke test for a repair agent.","caveats":"Bugs are hand-planted in toy functions, no repository context, and HumanEval derivatives are saturated and memorized."},{"id":"ifevalcode","name":"IFEvalCode","year":2025,"by":"Beihang University and others (Chai et al.)","workloads":["coding"],"task_types":["write correct code that also obeys explicit constraints on style, naming, algorithm or complexity","follow the same coding instruction given in Chinese or English"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["unit-tests","programmatic-check"],"size":null,"data":"synthetic","pii":"none","license":"unknown","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2507.22462","github":"https://ifevalcode.github.io/","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits coding assistants that must honor house style or constraints; borrow the split scoring of correctness versus instruction-following as two separate numbers.","caveats":"About 1.6K samples but the exact count and license are not on the abstract; constraints are checked by scripts that may not cover every phrasing; no repo context."},{"id":"livecodebench","name":"LiveCodeBench","year":2024,"by":"UC Berkeley, MIT, Cornell (Jain et al.)","workloads":["coding"],"task_types":["solve a LeetCode, AtCoder or Codeforces problem in Python","repair a wrong solution given test feedback","predict the output of code for a given input"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["unit-tests"],"size":1055,"data":"real-scraped","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2403.07974","github":"https://github.com/LiveCodeBench/LiveCodeBench","leaderboard":"https://livecodebench.github.io/leaderboard.html"},"good_for":["privacy-pattern","baseline"],"fit":"A model-level coding baseline rather than an agent test; borrow the date-windowed release scheme that lets you report only problems newer than a model's cutoff.","caveats":"Competitive programming only, no repository context or tools; scores vary with the date window chosen, so always state the release version (v6 has 1,055 problems)."},{"id":"livecodebench-pro","name":"LiveCodeBench Pro","year":2025,"by":"NYU and others (Zheng et al.)","workloads":["coding"],"task_types":["solve an Olympiad-level Codeforces, ICPC or IOI problem in C++","handle hard algorithmic reasoning and case analysis, not just implementation"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["unit-tests"],"size":null,"data":"real-scraped","pii":"none","license":"unknown","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.11928","github":"https://github.com/GavinZhengOI/LiveCodeBench-Pro","leaderboard":null},"good_for":["taxonomy","baseline"],"fit":"Fits teams that need a hard, uncontaminated algorithmic test with expert error labels; borrow the Olympiad-medalist failure taxonomy for grading reasoning errors.","caveats":"Problem count and license are not stated; leaderboard entry is by emailing results, so it is not independently run; algorithm puzzles, not software engineering."},{"id":"locobench-agent","name":"LoCoBench-Agent","year":2025,"by":"Salesforce AI Research","workloads":["coding"],"task_types":["answer questions and make changes in a codebase of 10K to 1M tokens over up to 50 turns","use file, search and code-analysis tools efficiently across ten languages"],"interaction":"multi-turn-scripted","environment":"filesystem","state_checked":false,"tools":["file operations","search","code analysis"],"grading":["programmatic-check","trajectory-check"],"size":8000,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2511.13998","github":"https://github.com/SalesforceAIResearch/LoCoBench-Agent","leaderboard":null},"good_for":["taxonomy","grading-design"],"fit":"Fits agents that must navigate large repositories; borrow its efficiency metrics such as files read versus files modified and read-before-write checks.","caveats":"Codebases are synthetically generated, so they lack real-world mess; nine metrics with no single headline number make comparisons across papers hard."},{"id":"long-code-arena","name":"Long Code Arena","year":2024,"by":"JetBrains Research","workloads":["coding","devops"],"task_types":["repair a failing GitHub Actions CI build from its logs","generate code that uses a specific library across a project","localize a bug, write a commit message, or summarize a module given the whole repo"],"interaction":"single-turn","environment":"mixed","state_checked":true,"tools":["none"],"grading":["programmatic-check","unit-tests","llm-judge"],"size":null,"data":"real-public","pii":"possible","license":"unknown","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2406.11612","github":"https://huggingface.co/spaces/JetBrains-Research/long-code-arena","leaderboard":"https://huggingface.co/spaces/JetBrains-Research/long-code-arena"},"good_for":["taxonomy","grading-design"],"fit":"Six project-wide tasks in one suite; the CI build repair task, graded by re-running the real GitHub Actions workflow, is a usable pattern for devops agents.","caveats":"Mostly single-turn model tasks rather than agent runs; per-task sizes and licenses were not visible from the paper page, and the leaderboard has not moved much since 2024."},{"id":"mint","name":"MINT","year":2023,"by":"University of Illinois Urbana-Champaign","workloads":["coding","general-assistant","tool-calling","embodied"],"task_types":["solve a math or reasoning problem by running Python","write a function and fix it after execution feedback","complete a household text-game task over several steps","revise an answer after natural-language feedback from a simulated user"],"interaction":"multi-turn-simulated-user","environment":"mixed","state_checked":false,"tools":["Python code execution","ALFWorld text environment"],"grading":["exact-match","unit-tests"],"size":586,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2309.10691","github":"https://github.com/xingyaoww/mint-bench","leaderboard":null},"good_for":["user-simulator","baseline"],"situations":["multi-turn","tool-error"],"fit":"Fits agents that iterate with a user on code or quantitative answers. The useful part is the GPT-4 feedback simulator and the turn-budget analysis (k = 1 to 5 turns), which shows how much a feedback loop is worth.","caveats":"Built from public datasets (GSM8K, HumanEval, MBPP, HotpotQA, ALFWorld) that are likely in training data; the repo has not been updated since 2024."},{"id":"multi-swe-bench","name":"Multi-SWE-bench","year":2025,"by":"ByteDance Seed","workloads":["coding"],"task_types":["resolve a GitHub issue in Java, TypeScript, JavaScript, Go, Rust, C or C++","compile and run a non-Python project's tests inside Docker"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests"],"size":1632,"data":"real-public","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2504.02605","github":"https://github.com/multi-swe-bench/multi-swe-bench","leaderboard":"https://multi-swe-bench.github.io/"},"good_for":["baseline","case-template","taxonomy"],"fit":"The largest multi-language issue set; fits coding agents that serve polyglot codebases. Borrow the 68-annotator screening rubric and the mini (400) and flash (300) subsets for cost.","caveats":"Language mix is uneven and compile-heavy languages make runs slow and expensive; results depend heavily on which scaffold (Agentless, SWE-agent, OpenHands) is used."},{"id":"projdevbench","name":"ProjDevBench","year":2026,"by":"Lu, Zhang et al. (zsworld6)","workloads":["coding"],"task_types":["build a complete C++ project from a requirements document","finish a partially written project so it passes online-judge test cases","design a system architecture that meets time and memory limits"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests","llm-judge","programmatic-check"],"size":20,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2602.01655","github":"https://github.com/zsworld6/projdevbench","leaderboard":null},"good_for":["grading-design"],"fit":"Fits agents that build whole programs end to end; borrow the 80/20 split between execution score and code review score and the rule-based checks for forbidden shortcuts.","caveats":"Only 20 problems, mostly C++ and judge-style rather than production code; the LLM review portion is not calibrated against humans."},{"id":"repobench","name":"RepoBench","year":2023,"by":"UC San Diego (Liu, Xu, McAuley)","workloads":["coding"],"task_types":["predict the next line of code using context from other files in the repo","retrieve the cross-file snippet needed to complete a line"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match","programmatic-check"],"size":null,"data":"real-public","pii":"none","license":"CC-BY-4.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2306.03091","github":"https://github.com/Leolty/repobench","leaderboard":null},"good_for":["grading-design"],"fit":"Fits autocomplete or inline-suggestion products, not agents; borrow exact match plus edit similarity plus CodeBLEU as cheap text-level grading for completion cases.","caveats":"Single-line completion only, Python and Java, no execution; data is from 2023 GitHub and likely in training sets."},{"id":"swe-bench","name":"SWE-bench","year":2023,"by":"Princeton NLP (Jimenez, Yang et al.)","workloads":["coding"],"task_types":["read a GitHub issue and edit the repository to resolve it","make a failing test pass without breaking other tests","coordinate changes across several functions or files"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests"],"size":2294,"data":"real-public","pii":"possible","license":"MIT","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2310.06770","github":"https://github.com/SWE-bench/SWE-bench","leaderboard":"https://www.swebench.com/"},"good_for":["grading-design","case-template","sandbox-design","baseline"],"fit":"Fits any agent that fixes bugs or implements changes in an existing codebase from a written issue. Borrow the FAIL_TO_PASS / PASS_TO_PASS test split and the per-task Docker image pattern.","caveats":"The full set has unclear or unsolvable issues (that is why Verified exists), is Python-only, and is in most training sets, so scores are contaminated; most teams now report Verified instead."},{"id":"swe-bench-java","name":"SWE-bench-java","year":2024,"by":"Multi-SWE-bench team (Zan et al.)","workloads":["coding"],"task_types":["resolve a GitHub issue in a Java project such as gson or jackson","make a failing JUnit test pass inside a Docker build"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests"],"size":91,"data":"real-public","pii":"possible","license":"unknown","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2408.14354","github":"https://github.com/multi-swe-bench/multi-swe-bench","leaderboard":null},"good_for":["case-template"],"fit":"An early Java-only issue set; useful if your agent targets JVM codebases and you want a tiny hand-verified pilot. Borrow the Docker-per-instance build setup for Maven projects.","caveats":"Only 91 instances from six repos, more than half from jackson-databind; folded into Multi-SWE-bench, which is the version to use now."},{"id":"swe-bench-lite","name":"SWE-bench Lite","year":2024,"by":"SWE-bench team (Princeton)","workloads":["coding"],"task_types":["fix a single-file bug described in a short GitHub issue","make a failing test pass without breaking other tests"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests"],"size":300,"data":"real-public","pii":"possible","license":"MIT","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2310.06770","github":"https://github.com/SWE-bench/SWE-bench","leaderboard":"https://www.swebench.com/lite.html"},"good_for":["case-template","baseline"],"fit":"A cheap smoke test for a coding agent: single-file edits, at most three hunks, no images or links in the issue. Borrow its filter rules when you need a small, clean subset of a bigger set.","caveats":"Filtered to easy single-file cases, so it over-estimates real ability; superseded by Verified for reporting and the leaderboard moves slowly."},{"id":"swe-bench-multilingual","name":"SWE-bench Multilingual","year":2025,"by":"SWE-bench team (Khandpur, Lieret, Jimenez, Press, Yang)","workloads":["coding"],"task_types":["resolve a GitHub issue in a C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby or Rust repo","build and run a project's own test suite in a language other than Python"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests"],"size":300,"data":"real-public","pii":"possible","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":null,"github":"https://github.com/SWE-bench/SWE-bench","leaderboard":"https://www.swebench.com/multilingual.html"},"good_for":["baseline","case-template"],"fit":"Fits coding agents that must work outside Python; 42 repos across 9 languages including redis, terraform, lucene and tokio. Use it to check that your harness is not Python-specific.","caveats":"No paper and only one published baseline (Claude 3.7 Sonnet with SWE-agent, 43 percent), so there is little to compare against; small per-language slices."},{"id":"swe-bench-multimodal","name":"SWE-bench Multimodal","year":2024,"by":"SWE-bench team (Princeton, Stanford)","workloads":["coding"],"task_types":["fix a visual bug in a JavaScript library from an issue that includes a screenshot","read an image in an issue or test and change front-end code accordingly"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor","image viewing"],"grading":["unit-tests"],"size":617,"data":"real-public","pii":"possible","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2410.03859","github":"https://github.com/SWE-bench/SWE-bench","leaderboard":"https://www.swebench.com/multimodal.html"},"good_for":["case-template","grading-design"],"fit":"Fits agents that work on front-end or visualization code where the bug report is a picture. Borrow the way image inputs are attached to an otherwise text-only task.","caveats":"JavaScript only, 17 libraries; the README says 480 tasks are runnable locally and the rest are a hidden split, so published numbers depend on which split was used."},{"id":"swe-bench-pro","name":"SWE-bench Pro","year":2025,"by":"Scale AI","workloads":["coding"],"task_types":["implement a multi-file change (10 to hundreds of lines) from an issue in a large repo","resolve an issue in Go, Python, JavaScript or TypeScript without breaking existing tests","work in a codebase the model is unlikely to have seen (copyleft or private)"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests"],"size":731,"data":"real-public","pii":"possible","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://static.scale.com/uploads/654197dc94d34f66c0f5184e/SWEAP_Eval_Scale%20(9).pdf","github":"https://github.com/scaleapi/SWE-bench_Pro-os","leaderboard":"https://labs.scale.com/leaderboard/swe_bench_pro_public"},"good_for":["baseline","privacy-pattern","case-template"],"fit":"Fits teams whose coding agent works on larger changes than SWE-bench; the held-out commercial split is the pattern to copy if you want a private test set that nobody can train on.","caveats":"Two versions (v1 with 731 tasks, v2 with 642 re-verified tasks) are not comparable; the commercial and held-out splits are private, and Inspect support is via the Harbor package."},{"id":"swe-bench-verified","name":"SWE-bench Verified","year":2024,"by":"OpenAI with the SWE-bench team","workloads":["coding"],"task_types":["read a GitHub issue and edit the repository to resolve it","make a failing test pass without breaking other tests","locate the right file and function in a large Python project"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests"],"size":500,"data":"real-public","pii":"possible","license":"MIT","harness":["inspect_evals","hal","own-repo"],"status":"active","links":{"paper":"https://openai.com/index/introducing-swe-bench-verified/","github":"https://github.com/SWE-bench/SWE-bench","leaderboard":"https://www.swebench.com/"},"good_for":["baseline","grading-design","case-template"],"fit":"The default number to compare against for issue-to-patch coding agents in Python. Borrow its human screening step, where engineers rejected tasks whose issue text or tests were unfair.","caveats":"Python only, 12 repos, and the tasks predate most current models, so leakage is likely; SWE-EVO reports frontier agents above 70 percent, so headroom is small. HAL runs a Mini subset, not all 500."},{"id":"swe-dev","name":"SWE-Dev","year":2025,"by":"Shanghai Jiao Tong University (Du et al.)","workloads":["coding"],"task_types":["implement a new feature in an existing Python codebase from a description","write about 190 lines across three files and pass developer-written unit tests"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests"],"size":500,"data":"real-public","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.16975","github":"https://github.com/DorothyDUUU/SWE-Dev","leaderboard":null},"good_for":["case-template","baseline"],"fit":"Fits feature-implementation agents rather than bug fixers. Borrow the Easy/Hard test split and the 14,000-instance training set if you train your own model.","caveats":"Python only; feature descriptions are derived from the merged code rather than written by a product owner, so they can leak the solution shape."},{"id":"swe-evo","name":"SWE-EVO","year":2025,"by":"FPT Software AI Center (Le, Thai, Bui et al.)","workloads":["coding"],"task_types":["move a Python project from one release to the next using the release notes as the spec","make coordinated changes across about 20 files while keeping hundreds of tests green"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests"],"size":48,"data":"real-public","pii":"possible","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2512.18470","github":"https://github.com/SWE-EVO/SWE-EVO","leaderboard":null},"good_for":["case-template","grading-design"],"fit":"Fits agents asked to do long multi-step work rather than one bug; borrow the partial-credit Fix Rate metric and the release-note-to-task construction.","caveats":"Only 48 tasks from seven Python projects, so one task swings results; frontier agents score around 25 percent, so it is hard but noisy."},{"id":"swe-gym","name":"SWE-Gym","year":2024,"by":"UC Berkeley, UIUC, CMU, Apple (Pan et al.)","workloads":["coding"],"task_types":["resolve a GitHub issue in one of 11 Python repos with a runnable environment","produce agent trajectories and verifier training data for coding agents"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests"],"size":2438,"data":"real-public","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2412.21139","github":"https://github.com/SWE-Gym/SWE-Gym","leaderboard":null},"good_for":["sandbox-design","case-template"],"fit":"A training environment rather than a test set; fits teams fine-tuning their own coding agent or verifier. Borrow the prebuilt per-instance Docker images and the Lite split of 234 tasks.","caveats":"Meant for training, so do not report it as a held-out score; Python only and overlaps in repo choice with SWE-bench."},{"id":"swe-lancer","name":"SWE-Lancer","year":2025,"by":"OpenAI","workloads":["coding","code-review"],"task_types":["implement a paid freelance feature or bug fix in the Expensify React Native app","pass hidden end-to-end browser tests written by professional engineers","choose the best of several engineers' implementation proposals as a manager"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor","browser for e2e tests"],"grading":["unit-tests","exact-match"],"size":1488,"data":"real-public","pii":"possible","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2502.12115","github":"https://github.com/openai/preparedness","leaderboard":null},"good_for":["grading-design","baseline","taxonomy"],"fit":"Fits product-engineering agents on a large JavaScript/TypeScript app, and manager-style agents that pick between proposals. Borrow dollar-weighted scoring and Playwright end-to-end tests.","caveats":"One codebase (Expensify) only; the public Diamond split is 502 tasks and the repo says only 198 of the 237 IC SWE Diamond tasks run offline; e2e tests are slow and flaky to run."},{"id":"swe-polybench","name":"SWE-PolyBench","year":2025,"by":"Amazon","workloads":["coding"],"task_types":["fix a bug, add a feature, or refactor code in a Java, JavaScript, TypeScript or Python repo","find the right files and syntax-tree nodes to change"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests","programmatic-check"],"size":2110,"data":"real-public","pii":"possible","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2504.08703","github":"https://github.com/amazon-science/SWE-PolyBench","leaderboard":"https://amazon-science.github.io/SWE-PolyBench/"},"good_for":["taxonomy","grading-design","baseline"],"fit":"Fits coding agents whose work mixes bug fixes, features and refactors; borrow the task-category labels and the file and node retrieval metrics as a partial-credit signal.","caveats":"Heavily skewed to JavaScript/TypeScript (about 1,700 of 2,110); use the PB500 or the 382-task Verified subset for balanced numbers."},{"id":"swe-rebench","name":"SWE-rebench","year":2025,"by":"Nebius","workloads":["coding"],"task_types":["resolve a freshly mined GitHub issue in a Python repo the model has not seen","set up an unfamiliar project's environment and run its tests"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests"],"size":null,"data":"real-public","pii":"possible","license":"CC-BY-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.20411","github":"https://huggingface.co/datasets/nebius/SWE-rebench","leaderboard":"https://swe-rebench.com/leaderboard"},"good_for":["privacy-pattern","baseline","case-template"],"fit":"Use it when you worry that SWE-bench scores are contaminated: the leaderboard is rebuilt monthly from new issues. Borrow the automated pipeline that turns issues into tested tasks.","caveats":"Tasks are filtered by LLM-written quality labels rather than humans, so some are unclear; the dataset is over 21,000 instances but the scored leaderboard subset changes each month."},{"id":"swe-smith","name":"SWE-smith","year":2025,"by":"Stanford, Princeton (Yang et al.)","workloads":["coding"],"task_types":["fix a synthetically injected bug in a real repo from a generated issue text","localize the file that a planted failure comes from"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests"],"size":52000,"data":"mixed","pii":"possible","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2504.21798","github":"https://github.com/SWE-bench/SWE-smith","leaderboard":null},"good_for":["case-template","privacy-pattern","sandbox-design"],"fit":"Fits teams that need many cheap coding cases from their own repository: point it at a repo, it breaks tests and writes issues. Borrow the bug-injection and keep-only-if-a-test-fails filter.","caveats":"Bugs and issue texts are synthetic, so they are easier and more uniform than real issues; a training toolkit, not a comparable leaderboard."},{"id":"usaco","name":"USACO","year":2024,"by":"Princeton NLP (Shi, Tang, Narasimhan, Yao)","workloads":["coding"],"task_types":["solve a USA Computing Olympiad problem in Python against hidden tests","use retrieval of similar problems or reflection to improve a solution"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["unit-tests"],"size":307,"data":"real-public","pii":"none","license":"unknown","harness":["inspect_evals","hal","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2404.10952","github":"https://github.com/princeton-nlp/USACO","leaderboard":"https://hal.cs.princeton.edu/"},"good_for":["baseline"],"fit":"A cost-tracked algorithmic baseline available on HAL; useful as a sanity check for reasoning-heavy coding agents, not for product engineering.","caveats":"Olympiad puzzles with public solutions online; no repository or tools; license is not stated in the repo."},{"id":"vibe-code-bench","name":"Vibe Code Bench","year":2026,"by":"Vals AI","workloads":["coding"],"task_types":["build and deploy a complete web application from a written specification","make every user workflow in the spec work when driven by a browser agent"],"interaction":"autonomous-long-horizon","environment":"mixed","state_checked":true,"tools":["shell","file editor","deployment"],"grading":["trajectory-check","llm-judge"],"size":100,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://caisconf.org/program/2026/papers/vibe-code-bench-evaluating-ai-models-on-end-to-end-web-application-development/","github":null,"leaderboard":"https://www.vals.ai/"},"good_for":["grading-design","user-simulator"],"fit":"Fits app-builder agents; borrow the idea of grading a deployed app by having a browser agent walk 964 scripted workflows instead of unit tests.","caveats":"Half the specs are held out and the code is not released, so you cannot run it yourself; several versions (v1, v1.1, 1-100) report very different top scores."},{"id":"crab","name":"CRAB","year":2024,"by":"CAMEL-AI (Xu et al.)","workloads":["computer-use","mobile","multi-agent"],"task_types":["complete a task that spans an Ubuntu desktop and an Android phone","move information between devices"],"interaction":"single-turn","environment":"mixed","state_checked":true,"tools":["Ubuntu VM","Android emulator","Python-wrapped actions"],"grading":["state-diff","trajectory-check"],"size":120,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2407.01511","github":"https://github.com/camel-ai/crab","leaderboard":null},"good_for":["grading-design","sandbox-design"],"fit":"Fits agents that must coordinate across a phone and a computer. Borrow the graph evaluator that gives partial credit for sub-goals reached in any valid order.","caveats":"Small; cross-device setup is complex; license not stated in the README."},{"id":"macosworld","name":"macOSWorld","year":2025,"by":"Show Lab, National University of Singapore","workloads":["computer-use","safety"],"task_types":["complete tasks in 30 macOS apps (system, productivity, media)","follow instructions in English, Chinese, Arabic, Japanese or Russian","resist on-screen deception attacks in the safety subset"],"interaction":"single-turn","environment":"desktop","state_checked":true,"tools":["cloud-hosted macOS instance","mouse and keyboard"],"grading":["programmatic-check"],"size":202,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.04135","github":"https://github.com/showlab/macosworld","leaderboard":null},"good_for":["sandbox-design","taxonomy"],"fit":"Fits agents that must run on Mac desktops or in several languages. Borrow the multilingual instruction and interface variants and the deception subset.","caveats":"Needs an AWS macOS instance and 15 to 20 minutes per task; grading details not confirmed from abstract or README; license not stated."},{"id":"omniact","name":"OmniACT","year":2024,"by":"Carnegie Mellon University / Writer (Kapoor et al.)","workloads":["computer-use","web-browsing"],"task_types":["write a PyAutoGUI script that performs a task from one screenshot","locate the right element to click, type or hotkey on macOS, Linux, Windows or web apps"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["single screenshot in","script out"],"grading":["programmatic-check"],"size":9802,"data":"real-scraped","pii":"none","license":"unknown","harness":["none"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2402.17553","github":null,"leaderboard":null},"good_for":["grading-design","taxonomy"],"fit":"Fits teams training grounding-plus-planning in one shot. Borrow the sequence score and action score (click-in-box, key set, typed-text BLEU).","caveats":"No execution, scripts are compared to a gold script; one screenshot per task so no multi-screen tasks; repo location and license not confirmed."},{"id":"osuniverse","name":"OSUniverse","year":2025,"by":"Kentauros AI","workloads":["computer-use","games"],"task_types":["click precisely on a desktop target","complete browser, terminal, GIMP, LibreOffice and multi-app tasks","play simple desktop games"],"interaction":"single-turn","environment":"desktop","state_checked":false,"tools":["Docker virtual desktop (AgentDesk)"],"grading":["llm-judge","human"],"size":160,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.03570","github":"https://github.com/agentsea/osuniverse","leaderboard":null},"good_for":["taxonomy","grading-design"],"fit":"Fits teams that want a difficulty ladder for desktop agents. Borrow the five tiers (paper to gold) with step budgets and the checklist-based Gemini validator.","caveats":"Validator is an LLM judge over screenshots, not a state check; small; difficulty tuned so agents stay under 50 percent at release."},{"id":"osworld","name":"OSWorld","year":2024,"by":"XLANG Lab (University of Hong Kong)","workloads":["computer-use","documents","spreadsheets","it-ops"],"task_types":["edit a document, spreadsheet or slide deck in LibreOffice","change settings or manage files in the OS","use Chrome, GIMP, VLC, VS Code or Thunderbird to finish a task","chain actions across several apps"],"interaction":"single-turn","environment":"desktop","state_checked":true,"tools":["mouse and keyboard on an Ubuntu VM","screenshots","accessibility tree"],"grading":["state-diff","programmatic-check"],"size":369,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["inspect_evals","own-repo"],"status":"deprecated","links":{"paper":"https://arxiv.org/abs/2404.07972","github":"https://github.com/xlang-ai/OSWorld","leaderboard":"https://os-world.github.io/"},"good_for":["sandbox-design","grading-design","case-template","baseline"],"situations":["unanswerable"],"fit":"The reference for desktop agents; borrow the per-task setup config plus execution-based evaluator script pattern and the VM snapshot reset.","caveats":"Original release had over 300 task and environment bugs and is superseded by OSWorld-Verified; VM hosting is heavy; Windows and macOS tasks are a small minority."},{"id":"osworld-2","name":"OSWorld 2.0","year":2026,"by":"XLANG Lab (University of Hong Kong)","workloads":["computer-use","documents","data-analysis","workflow-automation"],"task_types":["finish an hours-long professional workflow across several desktop apps","reason over input files and a stateful user profile","keep going through streaming or changing environments"],"interaction":"autonomous-long-horizon","environment":"desktop","state_checked":true,"tools":["Ubuntu VM","desktop apps","files","up to 500 steps"],"grading":["state-diff","programmatic-check"],"size":108,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2606.29537","github":"https://github.com/xlang-ai/OSWorld-V2","leaderboard":"https://os-world.github.io/"},"good_for":["baseline","case-template","grading-design"],"situations":["unanswerable"],"fit":"Fits long-running desktop agents; the successor to OSWorld-Verified with headroom. Borrow binary completion plus partial-credit scoring and the safety-sensitive action audit.","caveats":"Very expensive (about 318 tool calls per task); task data gated on Hugging Face; best score 20.6 percent binary; v2.1 is recommended over 2.0."},{"id":"osworld-human","name":"OSWorld-Human","year":2025,"by":"WukLab, UC San Diego (Abhyankar, Zhang)","workloads":["computer-use"],"task_types":["score an OSWorld run against the human reference number of steps","measure end-to-end latency and wasted actions"],"interaction":"single-turn","environment":"desktop","state_checked":true,"tools":["OSWorld result directories"],"grading":["trajectory-check"],"size":369,"data":"real-public","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.16042","github":"https://github.com/WukLab/osworld-human","leaderboard":null},"good_for":["grading-design"],"fit":"Fits any desktop agent team that must report latency or cost, not just success. Borrow the weighted efficiency score that combines success with steps relative to a human.","caveats":"Not a new task set, only human trajectories and a scorer for OSWorld; the repo warns it must not be used for training; license not stated."},{"id":"osworld-mcp","name":"OSWorld-MCP","year":2025,"by":"Tongyi Lab, Alibaba (Jia et al.)","workloads":["computer-use","tool-calling","documents"],"task_types":["finish an OSWorld-style desktop task with 158 MCP tools available","decide when to call a tool instead of clicking through the GUI"],"interaction":"single-turn","environment":"desktop","state_checked":true,"tools":["GUI actions","158 MCP tools across 7 apps"],"grading":["state-diff","programmatic-check","trajectory-check"],"size":null,"data":"mixed","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.24563","github":"https://github.com/X-PLUG/OSWorld-MCP","leaderboard":"https://osworld-mcp.github.io/"},"good_for":["grading-design","case-template"],"fit":"Fits agents that mix GUI control with API or MCP tools. Borrow the tool-invocation-rate and average-completion-steps metrics alongside accuracy.","caveats":"Task count not stated in abstract or site; tools were partly auto-generated; license not stated; numbers changed between preprint and ICLR versions."},{"id":"osworld-verified","name":"OSWorld-Verified","year":2025,"by":"XLANG Lab (University of Hong Kong)","workloads":["computer-use","documents","spreadsheets","it-ops"],"task_types":["the 369 OSWorld tasks with fixed evaluators, fuzzy matching and backup sites","office, daily and professional desktop tasks on Ubuntu"],"interaction":"single-turn","environment":"desktop","state_checked":true,"tools":["mouse and keyboard on an Ubuntu VM (AWS or Docker)","screenshots"],"grading":["state-diff","programmatic-check"],"size":369,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["inspect_evals","own-repo"],"status":"saturated","links":{"paper":"https://arxiv.org/abs/2404.07972","github":"https://github.com/xlang-ai/OSWorld","leaderboard":"https://os-world.github.io/"},"good_for":["baseline","grading-design","sandbox-design"],"situations":["unanswerable"],"fit":"Fits general desktop agents on office and OS tasks; the current reference number for computer-use models. Borrow the parallel AWS runner that finishes a run within an hour.","caveats":"Top scores (85 to 90 percent in 2026 reports) are past the 72 percent human baseline, so it no longer separates strong agents; use OSWorld 2.0 for headroom."},{"id":"screenspot-pro","name":"ScreenSpot-Pro (and ScreenSpot)","year":2025,"by":"National University of Singapore (Li et al.)","workloads":["computer-use"],"task_types":["click the element named by an instruction in a high-resolution professional app screenshot","ground targets in CAD, creative, developer, scientific and office software"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["screenshot in","coordinate out"],"grading":["exact-match"],"size":1581,"data":"real-scraped","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2504.07981","github":"https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding","leaderboard":"https://gui-agent.github.io/grounding-leaderboard"},"good_for":["baseline","case-template"],"fit":"Fits any screenshot-driven agent whose failures are mis-clicks; use it to test the grounding model alone. Borrow the instruction-plus-bounding-box case format.","caveats":"Grounding only, no task completion; the older ScreenSpot v1 set is near solved; 23 apps across 3 operating systems so coverage of any one app is thin."},{"id":"spider2-v","name":"Spider2-V","year":2024,"by":"xlang-ai (Ruisheng Cao et al.)","workloads":["computer-use","data-analysis","devops","sql"],"task_types":["complete a data engineering task inside a desktop app by clicking and typing","write SQL or Python in tools such as BigQuery, dbt, Airbyte, Dagster or Snowflake","follow tool documentation to configure a pipeline or warehouse job","finish a task in a remote cloud workspace through its web console"],"interaction":"autonomous-long-horizon","environment":"desktop","state_checked":true,"tools":["Ubuntu VM with pyautogui mouse and keyboard","20 data tools","browser","terminal"],"grading":["programmatic-check","state-diff"],"size":494,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2407.10956","github":"https://github.com/xlang-ai/Spider2-V","leaderboard":null},"good_for":["sandbox-design","grading-design","case-template"],"fit":"Fits agents that operate data tools through a GUI or mixed GUI plus code, such as a pipeline operator or warehouse admin assistant. Borrow its per-task evaluation functions and the VM built on OSWorld with human-validation scripts.","caveats":"Needs VMware and self-registered Snowflake and BigQuery accounts for part of the tasks; no public leaderboard, only 2024 baselines (GPT-4V 14.0%); heavy to set up and slow to run."},{"id":"ui-cube","name":"UI-CUBE","year":2025,"by":"UiPath","workloads":["computer-use","workflow-automation","crm"],"task_types":["perform a single UI interaction (click, select, type) across widget variants and resolutions","copy data between applications","complete an enterprise application scenario end to end"],"interaction":"single-turn","environment":"desktop","state_checked":true,"tools":["VNC desktop","X11 actions"],"grading":["state-diff","programmatic-check"],"size":226,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2511.17131","github":"https://github.com/UiPath/uipath_enterprise_benchmark","leaderboard":null},"good_for":["taxonomy","grading-design"],"fit":"Fits RPA-style enterprise agents; borrow the split into simple interactions versus multi-step workflows and the resolution and widget-variant sweep.","caveats":"Vendor-built; apps are synthetic stand-ins; repo has little documentation and no stated license; agents score 67 to 85 on simple tasks but 9 to 19 on workflows."},{"id":"weavebench","name":"WeaveBench","year":2026,"by":"Li, Shan et al. (arXiv 2606.09426)","workloads":["computer-use","coding","workflow-automation"],"task_types":["combine GUI actions with terminal and code edits in one long task","produce a deliverable file or artifact from a real user request"],"interaction":"autonomous-long-horizon","environment":"mixed","state_checked":true,"tools":["Ubuntu desktop plugin inside a CLI agent runtime","shell","code"],"grading":["llm-judge","trajectory-check","programmatic-check"],"size":114,"data":"real-public","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2606.09426","github":null,"leaderboard":null},"good_for":["grading-design"],"fit":"Fits coding agents that also need a GUI. Borrow the trajectory-aware judge that flags shortcuts such as faked screenshots or hard-coded results.","caveats":"Very new; no public repo found; judge-based grading; best pass rate 41.2 percent."},{"id":"windows-agent-arena","name":"Windows Agent Arena","year":2024,"by":"Microsoft","workloads":["computer-use","documents","it-ops"],"task_types":["change a Windows setting or manage files in Explorer","edit a document in LibreOffice, Notepad or VS Code","use Edge or Chrome, VLC, Paint, Clock or Calculator"],"interaction":"single-turn","environment":"desktop","state_checked":true,"tools":["mouse and keyboard on a Windows 11 VM","screenshots","UIA accessibility"],"grading":["state-diff","programmatic-check"],"size":154,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2409.08264","github":"https://github.com/microsoft/WindowsAgentArena","leaderboard":null},"good_for":["sandbox-design","baseline"],"fit":"Fits agents that must run on Windows desktops. Borrow the Azure-parallel runner that finishes the benchmark in about 20 minutes.","caveats":"Small; adapted from OSWorld so task style is similar; needs a Windows evaluation image; launch score 19.5 percent versus 74.5 for humans."},{"id":"crmarena","name":"CRMArena","year":2024,"by":"Salesforce AI Research","workloads":["crm","customer-support","knowledge-qa","tool-calling"],"task_types":["route a service case to the right agent given skills and workload","find the policy violation or the right knowledge article for a case","identify a named-entity or transfer-count metric from the CRM records","answer an analyst or manager question by querying linked CRM objects","say \"None\" when the question has no answer in the data"],"interaction":"single-turn","environment":"database","state_checked":false,"tools":["live Salesforce org through the Simple Salesforce API","SOQL and SOSL queries","record lookup tools"],"grading":["exact-match","programmatic-check"],"size":1170,"data":"synthetic","pii":"none","license":"CC-BY-NC-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2411.02305","github":"https://github.com/SalesforceAIResearch/CRMArena","leaderboard":"https://huggingface.co/spaces/Salesforce/CRMArena-Leaderboard"},"good_for":["sandbox-design","privacy-pattern","taxonomy","baseline"],"situations":["unanswerable","persona"],"fit":"Fits agents that read a real CRM (Salesforce or similar) to answer service, analyst or manager questions. Borrow the persona-based task taxonomy, the 30% unanswerable queries, and the way a synthetic org is generated from a schema with hidden latent variables that drive the ground truth.","caveats":"Single-turn and read-only (no writes to the CRM, no customer in the loop); non-commercial license; nine task types with 130 instances each; requires access to Salesforce orgs that the authors host, and the GUI access is no longer public."},{"id":"crmarena-pro","name":"CRMArena-Pro","year":2025,"by":"Salesforce AI Research","workloads":["crm","sales","customer-support","knowledge-qa","tool-calling"],"task_types":["resolve a service, sales or configure-price-quote request against B2B or B2C CRM records","clarify a vague request over several turns with a persona-driven user, then answer","execute a defined CRM workflow such as case routing or lead conversion","refuse to disclose confidential records when asked","answer a knowledge question from the org's articles"],"interaction":"multi-turn-simulated-user","environment":"database","state_checked":false,"tools":["live Salesforce B2B and B2C orgs through the Simple Salesforce API","SOQL and SOSL queries"],"grading":["exact-match","programmatic-check","llm-judge"],"size":4280,"data":"synthetic","pii":"none","license":"CC-BY-NC-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.18878","github":"https://github.com/SalesforceAIResearch/CRMArena","leaderboard":"https://huggingface.co/spaces/Salesforce/CRMArena-Leaderboard"},"good_for":["taxonomy","sandbox-design","privacy-pattern","baseline"],"situations":["ambiguous","forbidden","confidential","multi-turn","persona"],"fit":"Fits sales-assist and service agents that work inside a CRM, in both single-shot and conversational modes. Borrow the 19-task skill taxonomy (workflow execution, policy compliance, information retrieval, textual reasoning), the paired single-turn versus multi-turn scoring, and the confidentiality-awareness probes.","caveats":"Non-commercial license; the user simulator and answer extraction are both GPT-4o so results depend on that model; grading is answer matching rather than end-state checks; data is LLM-generated and only about two thirds of expert raters called it realistic."},{"id":"scuba","name":"SCUBA","year":2025,"by":"Salesforce AI Research","workloads":["crm","sales","customer-support","computer-use","web-browsing"],"task_types":["configure objects, fields, flows or permissions as a Salesforce admin","update leads, opportunities and accounts as a sales rep","handle cases and knowledge as a service agent","find information or troubleshoot inside the org"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["Salesforce developer org via browser (Playwright) or OSWorld-style desktop"],"grading":["state-diff","trajectory-check"],"size":300,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2509.26506","github":"https://github.com/SalesforceAIResearch/SCUBA","leaderboard":null},"good_for":["case-template","grading-design","sandbox-design"],"situations":["persona"],"fit":"Fits CRM agents working in Salesforce. Borrow the persona-based task sourcing from user interviews and milestone scoring against org snapshots.","caveats":"Needs a Salesforce developer org and OAuth login; single product; zero-shot top score 39 percent, about 50 with demonstrations."},{"id":"ecom-bench","name":"ECom-Bench","year":2025,"by":"Xiaoduo AI Lab","workloads":["customer-support","shopping","retail","tool-calling"],"task_types":["resolve an after-sales issue for a home or appliance product with a persona-driven shopper","look up the order and product information before answering","update the order or ticket state as the task requires","interpret an image the customer sends (18 of the tasks are multimodal)"],"interaction":"multi-turn-simulated-user","environment":"api-sandbox","state_checked":true,"tools":["e-commerce order and product lookup and update APIs over a local database"],"grading":["state-diff","programmatic-check","pass^k"],"size":53,"data":"mixed","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2507.05639","github":"https://github.com/XiaoduoAILab/ECom-Bench","leaderboard":null},"good_for":["user-simulator","grading-design","privacy-pattern"],"fit":"Fits e-commerce after-sales support agents. Borrow the persona files (consumption habits, personality) that drive the simulated shopper, and the three-part rule-based reward (end database state, required lookup tools called, required keywords in the reply) on top of the τ-bench harness.","caveats":"Only 53 tasks, so differences between models are noisy; built on τ-bench code; data was LLM-synthesized from historical commercial dialogues and the paper does not describe anonymization, so treat personal data as possible."},{"id":"enterprisebench-corecraft","name":"EnterpriseBench CoreCraft","year":2026,"by":"Surge AI","workloads":["customer-support","retail","tool-calling","workflow-automation"],"task_types":["resolve a customer support ticket for a PC parts company","issue or refuse a refund according to policy","look up orders, products and customers through tools","avoid exposing customer data in queries"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["23 MCP tools over a simulated company with 2","500+ entities"],"grading":["llm-judge"],"size":null,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2602.16179","github":null,"leaderboard":null},"good_for":["grading-design","sandbox-design"],"situations":["forbidden","confidential"],"fit":"Fits support agents built on MCP tools that must get every policy detail right. Borrow the all-or-nothing expert rubric scoring and the RL-environment framing.","caveats":"Built as a training environment by a data vendor; task count and release terms not public on the paper page; rubric grading is LLM-judged."},{"id":"intellagent","name":"IntellAgent","year":2025,"by":"Plurai","workloads":["customer-support","travel","education","tool-calling"],"task_types":["handle a generated edge-case customer scenario that combines several policies","act on a database through tools while staying within the stated rules","complete the conversation without violating any of the policies the scenario tests"],"interaction":"multi-turn-simulated-user","environment":"api-sandbox","state_checked":false,"tools":["the agent under test's own tools; example airline environment with a database"],"grading":["llm-judge"],"size":null,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2501.11067","github":"https://github.com/plurai-ai/intellagent","leaderboard":null},"good_for":["user-simulator","case-template","taxonomy"],"fit":"Fits any policy-bound chat agent whose team wants to generate their own test cases from a prompt or policy document. Borrow the policy graph (policies as nodes, co-occurrence and difficulty as edges) used to sample multi-policy scenarios, and the critique step that maps failures back to specific policies.","caveats":"A generator and framework, not a fixed dataset, so there is no stable number to compare against; grading is an LLM critic rather than a state check; optimization features are paid, and recent maintenance is unclear."},{"id":"journeybench","name":"JourneyBench","year":2026,"by":"Observe.AI","workloads":["customer-support","telecom","finance","shopping"],"task_types":["walk a customer through a multi-step support journey such as a loan application, order issue or telecom request","call the prescribed tools in the prescribed order with correct parameters","stay on the business-defined path when the user wanders"],"interaction":"multi-turn-simulated-user","environment":"api-sandbox","state_checked":false,"tools":["41 mocked tools with pre-generated responses"],"grading":["trajectory-check"],"size":703,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2601.00596","github":null,"leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits contact-center agents that must follow a business flow drawn as a graph. Borrow the User Journey Coverage Score (zero unless the tool-call sequence matches exactly, then the fraction of correct parameters) and the graph representation of journeys used to generate scenarios.","caveats":"No code or data released as of the paper; tools are mocked so there is no end state to check; sequence-exact scoring punishes valid alternative orderings; user is GPT-4o."},{"id":"sopbench","name":"SOPBench (customer service)","year":2025,"by":"Zekun Li et al.","workloads":["customer-support","finance","healthcare","government","tool-calling"],"task_types":["verify the user's identity and eligibility before taking a service action","follow a multi-step standard operating procedure for a bank, hotel, library, DMV, clinic, market or university request","call the right tools in the order the procedure requires","refuse an action whose constraints are not met, including under jailbreak pressure"],"interaction":"multi-turn-simulated-user","environment":"api-sandbox","state_checked":true,"tools":["167 domain functions across seven service domains"],"grading":["programmatic-check","trajectory-check"],"size":830,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2503.08669","github":"https://github.com/Leezekun/SOPBench","leaderboard":null},"good_for":["grading-design","case-template","taxonomy"],"situations":["forbidden"],"fit":"Fits support agents whose work is defined by written procedures with preconditions (check this, then do that). Borrow the idea of encoding each procedure as code and using it as the oracle verifier for outcome, step and trajectory compliance, so no LLM judge is needed.","caveats":"Task count differs between README (830) and abstract (900+); data license is CC-BY-4.0 while code is MIT; tasks and databases are LLM-generated; the user simulator is off by default so many reported runs are effectively single-user-turn."},{"id":"tau-bench","name":"τ-bench","year":2024,"by":"Sierra","workloads":["customer-support","retail","travel","tool-calling"],"task_types":["look up an order or flight reservation for a customer","exchange, return or cancel an order under the written retail policy","change or cancel a flight booking under the written airline policy","collect identity details from the user before acting","refuse a request the policy forbids"],"interaction":"multi-turn-simulated-user","environment":"api-sandbox","state_checked":true,"tools":["domain APIs over a JSON database (7 write and 8 read tools in retail, 6 write and 7 read in airline)"],"grading":["state-diff","pass^k"],"size":165,"data":"synthetic","pii":"none","license":"MIT","harness":["hal","own-repo"],"status":"deprecated","links":{"paper":"https://arxiv.org/abs/2406.12045","github":"https://github.com/sierra-research/tau-bench","leaderboard":null},"good_for":["grading-design","user-simulator","case-template"],"situations":["forbidden"],"fit":"Fits support or booking agents that must follow a written policy while talking to a customer and writing to a backend. Borrow the end-state database diff as the pass/fail check, the pass^k reliability metric, and the simulated user prompted with a hidden user instruction.","caveats":"The repo is marked deprecated and its tasks are not updated (dozens were later found invalid and fixed in the τ²-bench repo); only two domains, 115 retail and 50 airline tasks, all LLM-generated data, and the leaderboard is a static table in the README."},{"id":"tau-c-bench","name":"τᶜ-Bench (TASTE)","year":2026,"by":"Technion and IBM Research","workloads":["customer-support","telecom","retail","travel","tool-calling"],"task_types":["complete a support task that requires an unusual combination of domain tool calls","modify orders, bookings or telecom service under policy with a simulated user","handle tasks generated from a tool-call sequence rather than a written scenario"],"interaction":"multi-turn-simulated-user","environment":"api-sandbox","state_checked":true,"tools":["the τ²-bench airline","retail and telecom APIs"],"grading":["state-diff","pass^k"],"size":278,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2605.28556","github":"https://github.com/tomerkeren42/TASTE-task-synthesis-from-tool-sequence-evolution","leaderboard":null},"good_for":["case-template","taxonomy"],"situations":["forbidden","multi-turn"],"fit":"Fits teams whose agent already does well on τ²-bench and who need harder cases on the same tools. Borrow the method: sample valid tool sequences, cluster them for coverage, then write the task around the sequence and tune its difficulty.","caveats":"Harder extension of τ²-bench Verified (50 airline, 114 retail, 114 telecom tasks) rather than a new domain; the repo was not inspected here so license and runnability are unknown, and the tasks are LLM-generated and validated by LLM judges."},{"id":"tau2-bench","name":"τ²-bench","year":2025,"by":"Sierra","workloads":["customer-support","telecom","retail","travel","tool-calling"],"task_types":["troubleshoot a customer's phone service, mobile data or MMS problem by telling the user which steps to take on their device","look up and modify an order or flight reservation under policy","guide a user who holds their own tools through a multi-step fix","refuse or escalate when the policy forbids the request"],"interaction":"multi-turn-simulated-user","environment":"api-sandbox","state_checked":true,"tools":["agent-side domain APIs plus user-side device tools in telecom (dual control)"],"grading":["state-diff","programmatic-check","pass^k"],"size":279,"data":"synthetic","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.07982","github":"https://github.com/sierra-research/tau2-bench","leaderboard":"https://taubench.com"},"good_for":["grading-design","user-simulator","case-template","sandbox-design"],"situations":["forbidden","multi-turn","dual-control"],"fit":"Fits technical-support agents where the customer must act on their own device or account (dual control), and any policy-bound support agent that writes to a backend. Borrow the compositional task generator (atomic subtasks with init, solution and assertion functions), the assertion-based end-state check, and the error split between reasoning and communication.","caveats":"Size 279 is the sum of the paper's per-domain counts (115 retail, 50 airline, 114 telecom sampled from 2,285 generated); the repo has since moved on to τ³-bench and τ-bench 2.0 releases whose scores are not comparable with 1.x, and a 2026 TASTE paper reports frontier models near saturation on the verified task set."},{"id":"tau3-bench","name":"τ³-bench","year":2026,"by":"Sierra","workloads":["customer-support","finance","knowledge-qa","telecom","tool-calling"],"task_types":["answer a banking customer's question by retrieving the right policy document","handle airline, retail and telecom support tasks by voice in a full-duplex call","troubleshoot a telecom issue while guiding the user through device steps","modify an order or booking under policy"],"interaction":"multi-turn-simulated-user","environment":"mixed","state_checked":true,"tools":["domain APIs","document retrieval over 698 banking documents with 12 retrieval strategies","voice providers"],"grading":["state-diff","programmatic-check","trajectory-check","pass^k"],"size":null,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://sierra.ai/resources/research/tau-3-bench-knowledge","github":"https://github.com/sierra-research/tau2-bench","leaderboard":"https://taubench.com"},"good_for":["grading-design","user-simulator","sandbox-design"],"situations":["forbidden","multi-turn"],"fit":"Fits support agents that must answer from a knowledge base as well as act on a backend, and voice agents. Borrow the banking_knowledge domain pattern (retrieval graded against gold read calls) and the voice evaluation setup with simulated callers.","caveats":"Not a separate paper or repo: it is release v1.0.0 (March 2026) of the tau2-bench repo, adding a 97-task banking_knowledge domain, voice, and 75+ task fixes on top of the τ² domains; the grading was patched in 1.0.1 (scores moved up to 9 points), and the later τ-bench 2.0 release (multilingual, intake domain) again changed default simulator and judge models so numbers across releases are not comparable."},{"id":"autodcworkflow","name":"AutoDCWorkflow benchmark","year":2024,"by":"Lan Li et al.","workloads":["data-analysis","workflow-automation"],"task_types":["given a raw table and an analysis purpose, pick the columns that need cleaning","fix format inconsistencies, type errors and duplicates with OpenRefine operations","produce a minimal clean table that answers the stated purpose","output a cleaning workflow that matches a human-written one"],"interaction":"single-turn","environment":"api-sandbox","state_checked":true,"tools":["OpenRefine operations"],"grading":["programmatic-check","exact-match"],"size":142,"data":"real-public","pii":"possible","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2412.06724","github":null,"leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits data cleaning or data preparation agents that take a messy table and a goal. Borrow its three-way grading: does the clean table answer the purpose, how close are the cell values to the gold table, and how close is the operation sequence to a human workflow.","caveats":"No code repository was found; small (142 purposes over 96 tables, six topics); only six OpenRefine operations are covered; baselines are small open models; tables come from public datasets in social science, public health, finance and transportation."},{"id":"da-code","name":"DA-Code","year":2024,"by":"Yiming Huang et al.","workloads":["data-analysis","coding","ml-engineering"],"task_types":["load, clean and transform data files with Python or SQL (data wrangling)","run exploratory analysis, statistics and visualization on a dataset","train a classifier, regressor or clustering model and write predictions","plan multi-step code in a sandbox and write result files to a path"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":false,"tools":["Python","SQL and bash in a Docker sandbox"],"grading":["programmatic-check"],"size":500,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2410.07331","github":"https://github.com/yiyihum/da-code","leaderboard":"https://da-code-bench.github.io/"},"good_for":["sandbox-design","case-template","taxonomy"],"fit":"Fits data-analysis or notebook agents that write and run code over files. Borrow its three task categories (data wrangling, machine learning, exploratory analysis), the Docker sandbox and the evaluate.py comparison of output files to gold files.","caveats":"Leaderboard only lists 2024 models (best 30.5% with GPT-4) and has not been updated; the full task set is a separate download with only 100 examples in the repo; grading metrics are not described in the README."},{"id":"dabstep","name":"DABstep","year":2025,"by":"Adyen and Hugging Face","workloads":["data-analysis","finance","coding"],"task_types":["answer a factual question by combining CSV and JSON files with markdown documentation","read payment-processing rules in documentation and apply them to the data","write and run Python code over several files in multiple steps","return one exact value, string or list as the final answer"],"interaction":"single-turn","environment":"filesystem","state_checked":false,"tools":["Python code execution over local data files"],"grading":["exact-match","programmatic-check"],"size":450,"data":"synthetic","pii":"none","license":"CC-BY-4.0","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.23719","github":null,"leaderboard":"https://huggingface.co/spaces/adyen/DABstep"},"good_for":["grading-design","case-template","privacy-pattern"],"fit":"Fits finance or operations analytics agents that answer questions from files plus policy documents. Borrow its factoid grading without a judge model (numeric tolerance, fuzzy strings, element-wise lists) and its approach of making synthetic data with real-shaped business context.","caveats":"Ground truth is hidden and scored through a submission form; the payments data and business context are synthetic with inflated fraud rates; no dedicated GitHub repo, baseline code lives in a Hugging Face space; best hard-set score around 14-16%."},{"id":"datascibench","name":"DataSciBench","year":2025,"by":"THUDM (Tsinghua University)","workloads":["data-analysis","coding","science"],"task_types":["carry out an open-ended data science prompt that has no single right answer","produce code whose outputs are scored by named metrics","clean, analyse and visualise a dataset end to end"],"interaction":"single-turn","environment":"terminal","state_checked":false,"tools":["python code execution"],"grading":["programmatic-check"],"size":null,"data":"mixed","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2502.13897","github":"https://github.com/THUDM/DataSciBench","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits agents that take vague analysis requests rather than crisp questions. Borrow the Task-Function-Code idea: each prompt is tied to named aggregate functions and programmatic rules that score the execution result, plus a semi-automatic way to build ground truth.","caveats":"No task count on the abstract page or README, no license file in the repo, and ground truth is partly produced by LLM self-consistency, so it inherits model bias."},{"id":"ds-1000","name":"DS-1000","year":2022,"by":"xlang-ai (Lai et al.)","workloads":["data-analysis","coding"],"task_types":["complete a short Python snippet with pandas, numpy, scipy or scikit-learn","reshape, filter or clean a dataframe as a StackOverflow asker describes","produce a matplotlib plot that matches a reference","obey a surface constraint such as not using a named API"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["unit-tests","programmatic-check"],"size":1000,"data":"real-scraped","pii":"none","license":"CC-BY-SA-4.0","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2211.11501","github":"https://github.com/xlang-ai/DS-1000","leaderboard":null},"good_for":["grading-design","case-template","baseline"],"fit":"Fits a code assistant that writes pandas or numpy snippets for analysts. Borrow the two-part check (run on fresh inputs against a reference solution, plus a token check on the code text) and the practice of perturbing scraped problems to limit memorization.","caveats":"Single-shot code completion, not an agent loop; problems are from 2022 and library versions are pinned to that era; the README warns the test runner executes generated code and needs a sandbox."},{"id":"dsaeval","name":"DSAEval","year":2026,"by":"Hong Kong Polytechnic University (Sun et al.)","workloads":["data-analysis","ml-engineering","science"],"task_types":["carry out a multi-query data science project on one dataset","analyse unstructured data such as images or text as well as tables","deliver a notebook and a written report"],"interaction":"multi-turn-scripted","environment":"terminal","state_checked":false,"tools":["python code execution","image viewing"],"grading":["llm-judge"],"size":641,"data":"real-public","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2601.13591","github":null,"leaderboard":"https://dsaeval.lambda.org.ai/"},"good_for":["grading-design","taxonomy"],"fit":"Fits full data science agents that produce notebooks and reports over several requests. Borrow the three-part judge rubric (reasoning, code, results) with a consistency flag that checks the narrative against what the code produced.","caveats":"Grading is entirely by two LLM judges against a synthesized soft ground truth; no code repository or license was found; the paper says data will be released gradually."},{"id":"dsbench","name":"DSBench","year":2024,"by":"Jing et al. (LiqiangJing)","workloads":["data-analysis","ml-engineering","coding"],"task_types":["answer a numeric or multiple-choice question from a workbook with several tables","read a task brief that includes images and tables before analysing data files","train a model on a Kaggle dataset and submit predictions"],"interaction":"single-turn","environment":"terminal","state_checked":false,"tools":["python notebooks","data files"],"grading":["llm-judge","programmatic-check"],"size":540,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2409.07703","github":"https://github.com/LiqiangJing/DSBench","leaderboard":null},"good_for":["case-template","baseline","grading-design"],"fit":"Fits data science agents that must read long briefs and large or multi-table files, and agents that build models end to end. Borrow the relative performance gap metric that scores a model against the human leaderboard on the same Kaggle task.","caveats":"The analysis answer checker needs an OpenAI key, so grading is model-assisted; the README says the dataset is for non-commercial use even though the code is MIT; the modeling part has only 74 tasks."},{"id":"dseval","name":"DSEval","year":2024,"by":"Microsoft Research (Zhang et al.)","workloads":["data-analysis","coding"],"task_types":["solve a sequence of dependent data science steps in one session","answer a StackOverflow-style pandas question with code","work through a Kaggle notebook as a chain of problems"],"interaction":"multi-turn-scripted","environment":"terminal","state_checked":false,"tools":["python code execution"],"grading":["programmatic-check"],"size":825,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2402.17168","github":"https://github.com/MetaCopilot/dseval","leaderboard":null},"good_for":["case-template","grading-design"],"fit":"Fits notebook-style assistants where each question builds on the last. Borrow the problem set format (ordered problems sharing one session) and the bootstrapped annotation method for expanding cases.","caveats":"Four sub-benchmarks of uneven size and difficulty; the README does not describe the validators; sources include LeetCode and StackOverflow, which models have seen."},{"id":"infiagent-dabench","name":"InfiAgent-DABench","year":2024,"by":"InfiAgent team (Hu et al.)","workloads":["data-analysis","coding"],"task_types":["answer a question about a CSV file by writing and running Python","compute summary statistics, correlations or filtered counts from a table","return the result in a fixed closed-form answer format"],"interaction":"single-turn","environment":"terminal","state_checked":false,"tools":["python code execution"],"grading":["exact-match"],"size":257,"data":"real-public","pii":"none","license":"CC-BY-NC-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2401.05507","github":"https://github.com/InfiAgent/InfiAgent","leaderboard":null},"good_for":["grading-design","case-template","baseline"],"fit":"Fits a data analysis assistant that answers questions over uploaded CSV files with code. Borrow the trick of rewriting open-ended questions into closed-form answers so grading is an exact match.","caveats":"Only the validation split is public; grading first reformats the agent's reply with a GPT-3.5 call, and the data license is non-commercial. Small and from early 2024."},{"id":"kramabench","name":"KramaBench","year":2025,"by":"MIT CSAIL","workloads":["data-analysis","science","coding","research"],"task_types":["find the relevant files among about 1,700 raw files in a data lake","clean and join heterogeneous files from several sources","write Python that computes a numeric or text answer to a domain question","break a question into sub-tasks and solve each one"],"interaction":"single-turn","environment":"filesystem","state_checked":false,"tools":["Python code execution over local files"],"grading":["programmatic-check","unit-tests","llm-judge"],"size":104,"data":"real-public","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.06541","github":"https://github.com/mitdbg/KramaBench","leaderboard":null},"good_for":["grading-design","taxonomy","case-template"],"fit":"Fits analytics agents that work over many raw files rather than one tidy database. Borrow its sub-task decomposition (633 sub-tasks under 104 tasks), its mix of exact metrics, generated unit tests and rubric partial credit, and its six domain split.","caveats":"No license file in the repo and the dataset card only says \"cc\"; headline accuracy differs between paper versions (55% versus 50%); scientific domains only (archaeology, astronomy, biomedical, environment, legal, wildfire); top repo score 22%."},{"id":"longda","name":"LongDA","year":2026,"by":"University of Notre Dame and IBM Research (Li et al.)","workloads":["data-analysis","documents","research"],"task_types":["find variable definitions and rules in long survey documentation","combine facts from several documents before computing","write and run code over raw survey files to produce a number or list"],"interaction":"single-turn","environment":"mixed","state_checked":false,"tools":["document search and reading","python code execution"],"grading":["programmatic-check"],"size":505,"data":"real-public","pii":"possible","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2601.02598","github":null,"leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits analytics agents whose hard part is reading documentation, such as codebooks or data dictionaries, before querying data. Borrow the two metrics: answer structure coverage and numeric match within a tolerance.","caveats":"No code repository or license was found; the agent tool set is prescribed by the authors' own framework; survey microdata may carry respondent-level records."},{"id":"magellan-er","name":"Magellan / DeepMatcher entity matching datasets","year":2018,"by":"AnHai Doan group (University of Wisconsin-Madison)","workloads":["data-analysis"],"task_types":["decide whether two records from different tables refer to the same entity","match citations, products, restaurants, songs or companies across sources","handle dirty records where attribute values are moved into the title field","match text-only records such as company pages and homepages"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["programmatic-check"],"size":null,"data":"real-public","pii":"none","license":"unknown","harness":["none"],"status":"frozen","links":{"paper":null,"github":"https://github.com/anhaidgroup/deepmatcher/blob/master/Datasets.md","leaderboard":null},"good_for":["case-template","baseline","taxonomy"],"fit":"Fits record linkage or deduplication agents that need labeled pairs to test on across domains. Borrow its three flavors (structured, textual, dirty) and its 3:1:1 splits as a template for building a matching test set.","caveats":"Classic ML datasets (13 sets from 450 to 112,632 labeled pairs), not an agent benchmark: no tools, no environment, F1 on pairs; most sets date from 2015-2018 and are close to saturated for fine-tuned models; no license is stated on the datasets page."},{"id":"opensanctions-pairs","name":"OpenSanctions Pairs","year":2026,"by":"Chandler Smith et al. (OpenSanctions)","workloads":["data-analysis","finance","government","security"],"task_types":["decide whether two sanctions or watchlist records refer to the same person or company","match names across languages and writing systems","compare records with partial dates, aliases and identifiers"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["programmatic-check"],"size":755540,"data":"real-public","pii":"yes","license":"unknown","harness":["none"],"status":"saturated","links":{"paper":"https://arxiv.org/abs/2603.11051","github":null,"leaderboard":null},"good_for":["baseline","case-template"],"fit":"Fits compliance screening or know-your-customer agents that match entities against sanctions lists. Borrow its pairwise labeled format from 293 sources across 45 jurisdictions and its rule-based baseline (91.3% F1) as a floor.","caveats":"Pairwise matching is already near ceiling (GPT-4o 99.0% F1) so it will not separate models; records are real named persons and organizations; no dataset or code link was found on the abstract page and the data license is not stated there."},{"id":"qrdata","name":"QRData","year":2024,"by":"Liu et al. (xxxiaol)","workloads":["data-analysis","science","education"],"task_types":["answer a statistics question using an attached data sheet","answer a causal reasoning question using the provided data","give a multiple-choice or numeric answer"],"interaction":"single-turn","environment":"terminal","state_checked":false,"tools":["python code interpreter"],"grading":["exact-match"],"size":411,"data":"real-public","pii":"none","license":"CC-BY-NC-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2402.17644","github":"https://github.com/xxxiaol/QRData","leaderboard":null},"good_for":["case-template","baseline"],"fit":"Fits analysis assistants that must apply statistics or causal methods correctly, not just run code. Borrow the pairing of each question with a data sheet and a companion text-only set to tell reasoning failures from data-handling failures.","caveats":"Non-commercial license; the README admits measurement errors in some text-only numeric questions; questions come from textbooks and papers, so they may be in training data."},{"id":"realhitbench","name":"RealHiTBench","year":2025,"by":"Zhejiang University and vivo (Wu et al.)","workloads":["data-analysis","knowledge-qa","documents"],"task_types":["answer a question over a table with hierarchical row or column headers","read a table given as LaTeX, HTML or an image","reason across nested sub-tables or joined tables"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["programmatic-check"],"size":3752,"data":"real-public","pii":"none","license":"CC-BY-NC-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.13405","github":"https://github.com/cspzyy/RealHiTBench","leaderboard":null},"good_for":["case-template","taxonomy","baseline"],"fit":"Fits assistants that read reports or financial tables with multi-level headers. Borrow the structure taxonomy (hierarchical headers, nested sub-tables, multi-table joins, decorative elements) and the practice of testing the same table in several input formats.","caveats":"Non-commercial data license; the README does not explain the scoring metrics; read-only question answering with no table editing."},{"id":"tablebench","name":"TableBench","year":2024,"by":"Wu et al. (TableBench)","workloads":["data-analysis","knowledge-qa"],"task_types":["check whether a claim is supported by a table","compute a number from a table","find a trend, correlation or statistic in a table","write code that draws a chart from a table"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["optional python interpreter"],"grading":["exact-match","programmatic-check"],"size":886,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2408.09174","github":"https://github.com/TableBench/TableBench","leaderboard":"https://tablebench.github.io/"},"good_for":["taxonomy","grading-design","baseline"],"fit":"Fits assistants that answer questions over a single table in a chat. Borrow the 18-category taxonomy under four headings and the per-category metric choice (exact match, exact match with a 10 percent margin, ROUGE-L, pass at 1 for charts).","caveats":"Questions were generated by GPT-4 agents from seed examples and only 30 percent were human-checked; tables are reused from older datasets such as WTQ and TabFact, so leakage is possible; ROUGE-L is a weak check for open-ended answers."},{"id":"wdc-products","name":"WDC Products","year":2022,"by":"Web Data Commons (University of Mannheim)","workloads":["data-analysis","retail","shopping"],"task_types":["decide whether two product offers from different shops describe the same product","match offers with hard corner cases such as near-identical model numbers","generalize to products never seen in the training data","classify an offer into one of many product entities (multi-class form)"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["programmatic-check"],"size":4500,"data":"real-scraped","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2301.09521","github":null,"leaderboard":"https://webdatacommons.org/largescaleproductcorpus/wdc-products/"},"good_for":["taxonomy","baseline","case-template"],"fit":"Fits product catalog or deduplication agents that must say whether two records are the same item. Borrow its difficulty dials (share of corner cases, share of unseen entities, training set size) across 27 variants built from 11,715 offers of 2,162 products.","caveats":"A classic supervised entity matching dataset, not an agent benchmark: F1 on labeled pairs with no tools or environment; size above is pairs per pairwise test set; data is scraped from 3,259 shops via Common Crawl in 2020 and no license is stated."},{"id":"androidbuildbench","name":"AndroidBuildBench","year":2025,"by":"Son, Ren, Liu, Zhao","workloads":["devops","coding","mobile"],"task_types":["repair a broken Gradle build in an open-source Android project","resolve dependency and toolchain errors until the project compiles"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","gradle"],"grading":["programmatic-check"],"size":1019,"data":"real-public","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.08640","github":null,"leaderboard":null},"good_for":["case-template"],"fit":"Fits build-and-dependency repair agents; its \"tool bridging\" result (domain tools beat a raw shell) is a design lesson for any devops agent.","caveats":"No code link on the paper page, license unknown, and the authors' own agent is at 81 percent so it is close to saturation for that setup."},{"id":"ci-bench","name":"CI-Bench","year":2026,"by":"UC Davis (Nabil, Zhu, Rubio-Gonzalez)","workloads":["devops","coding"],"task_types":["read a failed CI build log and patch the code so the build passes","re-run the fix in the original BugSwarm container environment"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["programmatic-check","unit-tests"],"size":100,"data":"real-public","pii":"possible","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://conf.researchr.org/details/icse-2026/icse-2026-demonstrations/9/CI-Bench-A-Framework-for-Evaluating-Large-Language-Model-Tools-on-CI-Failures","github":"https://github.com/BugSwarm/CI-Bench","leaderboard":null},"good_for":["sandbox-design","grading-design"],"fit":"Fits CI-repair or build-fixing agents; borrow the log-to-prompt pipeline and the re-execution in the exact failing container as the pass check.","caveats":"Only 100 failures, a demo-track paper, no license in the repo, and best tool scored 36 percent with 2025-era models."},{"id":"dataclaweval","name":"DataClawEval","year":2026,"by":"Debin Meng et al.","workloads":["devops","sql","coding","data-analysis"],"task_types":["write a batch job in PySpark, HiveSQL, MySQL or Trino from a business request","write a streaming FlinkSQL job","explore the schema and verify the output before finishing","produce output tables with the right schema, rows and numbers"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["PySpark","HiveSQL","MySQL","Trino","FlinkSQL in a fresh Docker container","shell"],"grading":["programmatic-check","state-diff","trajectory-check"],"size":100,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2607.28033","github":"https://github.com/Dicemy/DataClawEval","leaderboard":null},"good_for":["grading-design","sandbox-design"],"fit":"Fits data engineering agents that write jobs for several engines and must verify their own output. Borrow its deterministic rule-based graders and its score that blends artifact quality with process quality (exploration, efficiency, self-verification).","caveats":"Very new (July 2026) with scores already around 75 so headroom is limited; half the prompts are in Chinese; the harness is wired to specific coding-agent backends (Claude Code, Codex, CodeBuddy); process points reward behavior not just results."},{"id":"elt-bench","name":"ELT-Bench","year":2025,"by":"University of Illinois Urbana-Champaign (Daniel Kang group)","workloads":["devops","sql","workflow-automation","coding"],"task_types":["set up Airbyte connectors to extract from APIs, S3, PostgreSQL, MongoDB and flat files","load source tables into a cloud warehouse such as Snowflake","write dbt SQL models that produce the required data models","debug a long pipeline run until the output tables match"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["Airbyte","dbt","Terraform","Snowflake or Databricks or Redshift","Docker","shell"],"grading":["state-diff","programmatic-check"],"size":100,"data":"mixed","pii":"none","license":"CC-BY-SA-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2504.04808","github":"https://github.com/uiuc-kang-lab/ELT-Bench","leaderboard":null},"good_for":["sandbox-design","grading-design","case-template"],"fit":"Fits data engineering agents that build or maintain extract-load-transform pipelines. Borrow its two-stage grading that compares produced warehouse tables to ground-truth CSVs and its containerized source systems (835 source tables, 203 data models).","caveats":"Needs a paid warehouse account and Airbyte deployment; best result in the paper was 3.9% of data models correct at about $4.30 per pipeline; the current repo (ELT-Bench++) has shifted tooling and its README no longer mentions dbt or Terraform; data provenance is not stated in the abstract or README."},{"id":"iac-eval","name":"IaC-Eval","year":2024,"by":"University of Michigan (Kon et al.)","workloads":["devops","it-ops"],"task_types":["write Terraform for an AWS setup described in plain language","produce a configuration that satisfies a policy-as-code intent check"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["terraform","opa"],"grading":["programmatic-check"],"size":458,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://www.cs-pk.com/neurips24-iac-eval.pdf","github":"https://github.com/autoiac-project/iac-eval","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits infrastructure-as-code agents; borrow intent specs checked with OPA against the Terraform plan instead of string-matching the HCL.","caveats":"AWS and Terraform only, single-turn, and the README calls it not production-ready; GPT-4 scored 19 percent at release so it may be partly saturated by now."},{"id":"multi-iac-bench","name":"Multi-IaC-Bench","year":2025,"by":"Amazon Web Services (Davidson et al.)","workloads":["devops","it-ops"],"task_types":["modify an existing CloudFormation, Terraform or CDK template per a change request","keep the template syntactically valid and semantically aligned with the request"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["iac validators"],"grading":["programmatic-check","llm-judge"],"size":null,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2509.05303","github":null,"leaderboard":null},"good_for":["case-template"],"fit":"Fits agents that edit infrastructure templates rather than write them fresh; the (initial template, change request, target template) triplet is the case format to copy.","caveats":"Synthetic, no public code link on the paper page, task count unstated, and models are already above 95 percent on syntax so only semantic checks separate them."},{"id":"terminal-bench-1","name":"Terminal-Bench 1.0 (Core v0.1)","year":2025,"by":"Stanford and Laude Institute","workloads":["devops","coding","it-ops"],"task_types":["complete a sysadmin or build task in a Linux shell such as compiling code or configuring a server","train a small model or process data from the command line","leave the container in a state that a hidden test script accepts"],"interaction":"autonomous-long-horizon","environment":"terminal","state_checked":true,"tools":["shell"],"grading":["programmatic-check","state-diff"],"size":null,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"deprecated","links":{"paper":"https://arxiv.org/abs/2601.11868","github":"https://github.com/laude-institute/terminal-bench","leaderboard":"https://www.tbench.ai/"},"good_for":["sandbox-design","grading-design","case-template"],"fit":"The original terminal task format (task.yaml, Dockerfile, solution.sh, test script) is a clean template for any shell-based agent case; borrow the oracle-solution check.","caveats":"Replaced by 2.0 and later versions; the beta had roughly 100 tasks of uneven quality and the site now reports only the newest version, so old scores are not comparable."},{"id":"terminal-bench-2","name":"Terminal-Bench 2.0","year":2025,"by":"Stanford and Laude Institute (Merrill et al.)","workloads":["devops","coding","it-ops"],"task_types":["complete a hard real-world task in a Linux shell such as fixing a build, setting up a service or recovering data","write and run code or scripts from the command line until a hidden test passes","manage long multi-step work in a container without a user"],"interaction":"autonomous-long-horizon","environment":"terminal","state_checked":true,"tools":["shell"],"grading":["programmatic-check","state-diff"],"size":89,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2601.11868","github":"https://github.com/laude-institute/terminal-bench","leaderboard":"https://www.tbench.ai/"},"good_for":["baseline","sandbox-design","grading-design","case-template"],"fit":"The standard number for terminal and devops agents; frontier agents were under 65 percent at release. Borrow Harbor's container-per-task layout and the human-written oracle solution.","caveats":"Only 89 tasks so confidence intervals are wide; the project has since moved to newer versions (the site now shows 4.0) and a 2.1 release could not be confirmed, so pin the version."},{"id":"cord","name":"CORD","year":2019,"by":"NAVER CLOVA","workloads":["documents","retail"],"task_types":["extract menu items, prices, subtotal and total from a receipt photo","group extracted fields into line items","parse a receipt after OCR into a structured record"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["programmatic-check"],"size":1000,"data":"real-public","pii":"possible","license":"CC-BY-4.0","harness":["none"],"status":"frozen","links":{"paper":null,"github":"https://github.com/clovaai/cord","leaderboard":null},"good_for":["case-template","baseline"],"fit":"Fits receipt and expense capture agents. Borrow its two-level label scheme (five superclasses, 42 subclasses) and its line-group annotations as a template for structured extraction targets.","caveats":"From 2019 and small (1,000 Indonesian receipts, 100 in test); the repo ships no scorer, so field-level scores across papers are not directly comparable; essentially solved by current document models and useful mostly as a smoke test."},{"id":"docbench","name":"DocBench","year":2024,"by":"Zou et al. (Tencent AI Lab / Shanghai Jiao Tong University)","workloads":["documents","knowledge-qa","finance","legal"],"task_types":["answer a question about a long real document (PDF)","extract a number or table value from a document","reason over several parts of one document","handle questions that cannot be answered from the document"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["document file as input"],"grading":["llm-judge"],"size":1102,"data":"real-public","pii":"possible","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2407.10701","github":"https://github.com/Anni-Zou/DocBench","leaderboard":null},"good_for":["case-template","taxonomy"],"fit":"Fits document question-answering assistants over contracts, filings or papers. Borrow the four question types (including unanswerable) over 229 real documents in five domains.","caveats":"Read-only; grading relies on an OpenAI judge; no explicit data license; paper marked work in progress."},{"id":"docile","name":"DocILE","year":2023,"by":"Rossum","workloads":["documents","finance","workflow-automation"],"task_types":["locate and extract key fields (amounts, dates, identifiers) from an invoice or order","extract each line item from a table on a business document","handle a document layout never seen in training"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["programmatic-check"],"size":6700,"data":"mixed","pii":"possible","license":"MIT","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2302.05658","github":"https://github.com/rossumai/docile","leaderboard":null},"good_for":["grading-design","case-template","taxonomy"],"fit":"Fits invoice and purchase-order extraction agents. Borrow its 55-class field taxonomy, its line-item grouping task, and its scorer that matches predicted fields to ground truth by bounding-box overlap and reports precision, recall and F1 per field.","caveats":"The 2023 competition has ended; the data download needs a token from docile.rossum.ai and the dataset license is not stated in the paper; only 6.7k annotated documents, padded with 100k synthetic ones; built for layout models, not for agent workflows."},{"id":"docvqa","name":"DocVQA","year":2020,"by":"Computer Vision Center (UAB) and IIIT Hyderabad","workloads":["documents"],"task_types":["answer a question about a scanned document page from its image","read a value out of a form, table or letter","use the page layout to find where an answer sits"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["programmatic-check"],"size":50000,"data":"real-public","pii":"possible","license":"unknown","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2007.00398","github":null,"leaderboard":"https://www.docvqa.org/"},"good_for":["baseline","case-template"],"fit":"Fits agents that read scanned or photographed documents (forms, letters, reports) and pull out a short answer. Borrow its question-over-image case format and its ANLS scoring, which tolerates small OCR spelling errors instead of demanding an exact string.","caveats":"From 2020 and older than most of this index, though still widely reported; human accuracy is 94% and current vision models are close to it, test-split answers are held back, and the dataset license is not stated."},{"id":"enterpriseclawbench","name":"EnterpriseClawBench","year":2026,"by":"Frontis AI","workloads":["documents","data-analysis","workflow-automation","general-assistant"],"task_types":["reproduce a real workplace agent session from its recovered files","read mixed files and deliver a business artifact (report, sheet, deck)","call tools in a sandbox to finish the job"],"interaction":"single-turn","environment":"containers","state_checked":false,"tools":["sandbox with files and tools; bring your own harness"],"grading":["llm-judge","programmatic-check"],"size":852,"data":"real-scraped","pii":"possible","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2606.23654","github":"https://github.com/FrontisAI/EnterpriseClawBench","leaderboard":null},"good_for":["privacy-pattern","taxonomy","grading-design"],"fit":"Fits teams who have their own agent logs and want to turn them into evals. Borrow the construction pipeline (fixtures recovered, prompts rewritten, rubrics and rules generated) and the harness-plus-model reporting.","caveats":"The task data itself is private; only the protocol and one sanitized example are public; judge-graded."},{"id":"gdpval","name":"GDPval","year":2025,"by":"OpenAI","workloads":["documents","spreadsheets","finance","legal","general-assistant"],"task_types":["produce a professional deliverable (report, brief, plan, deck) from a brief and reference files","build or fix a spreadsheet model","draft legal, nursing, engineering or sales documents","review and summarize supplied documents"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none required; file inputs and outputs"],"grading":["human","llm-judge"],"size":220,"data":"real-public","pii":"none","license":"unknown","harness":["inspect_evals"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.04374","github":"https://huggingface.co/datasets/openai/gdpval","leaderboard":null},"good_for":["grading-design","case-template","taxonomy"],"fit":"Fits document-producing assistants in knowledge-work roles. Borrow the occupation-by-task taxonomy (44 occupations, 9 sectors) and the weighted rubrics written by professionals.","caveats":"The main metric is human pairwise judgment; only the 220-task gold set is public (1,320 total); no tool use or environment, so it does not test acting in systems."},{"id":"longbench-v2","name":"LongBench v2","year":2024,"by":"Tsinghua University (THUDM)","workloads":["documents","knowledge-qa","coding","research"],"task_types":["answer a multiple-choice question about one long document","answer a question that needs facts from several long documents","recall a detail from a very long dialogue history","answer a question about a whole code repository","read a long structured file such as a table or log"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match"],"size":503,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2412.15204","github":"https://github.com/THUDM/LongBench","leaderboard":"https://longbench2.github.io/#leaderboard"},"good_for":["baseline","case-template","taxonomy"],"fit":"Fits any agent that must read long inputs (8k to 2M words) before answering, such as contract review or codebase question answering. Borrow its six-category task split and its human-written, time-limited question style, which keeps questions hard to shortcut.","caveats":"Multiple choice only, so it does not test free-form answers or extraction; 503 questions across six categories leaves each category small; human experts score only 53.7% under a 15-minute limit, so the ceiling is unclear."},{"id":"officebench","name":"OfficeBench","year":2024,"by":"Wang et al. (UC San Diego)","workloads":["documents","spreadsheets","email-calendar","general-assistant"],"task_types":["read a document or PDF and write the answer into a file","edit spreadsheet cells and convert to PDF","create or delete calendar events","read and send email","switch between apps to finish one office task"],"interaction":"single-turn","environment":"containers","state_checked":true,"tools":["Word","Excel","PDF","Calendar","Email","OCR","Shell operations in Docker"],"grading":["exact-match","programmatic-check"],"size":300,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2407.19056","github":"https://github.com/zlwang-cs/OfficeBench","leaderboard":null},"good_for":["case-template","grading-design","sandbox-design"],"fit":"Fits office assistants that move content between documents, spreadsheets, email and calendars. Borrow the three-tier design (one, two, three apps) and the mix of exact, fuzzy and execution-based checks on output files.","caveats":"Apps are thin Python wrappers rather than real Office software; tasks are small and the best models already pass the one- and two-app tiers."},{"id":"omnidocbench","name":"OmniDocBench","year":2024,"by":"Shanghai AI Laboratory (OpenDataLab)","workloads":["documents"],"task_types":["convert a PDF page to markdown with correct reading order","extract a table from a page as structured markup","transcribe formulas from a page","parse hard layouts such as newspapers, handwriting and financial reports"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["programmatic-check"],"size":1651,"data":"real-public","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2412.07626","github":"https://github.com/opendatalab/OmniDocBench","leaderboard":"https://github.com/opendatalab/OmniDocBench"},"good_for":["grading-design","baseline","taxonomy"],"fit":"Fits the parsing step of any document pipeline, where PDFs become text for an agent to reason over. Borrow its per-element metrics (edit distance for text, TEDS for tables, CDM for formulas) and its page attribute labels for slicing results by layout and language.","caveats":"Tests parsing only, not question answering or extraction of business fields; the dataset is research-use only even though the code is Apache-2.0; version labels in the README are inconsistent (v1.6 leaderboard, v1.7 data); 1,651 pages."},{"id":"edubench","name":"EduBench","year":2025,"by":"Beijing Institute of Technology (Xu, Huang et al.)","workloads":["education","knowledge-qa","documents"],"task_types":["answer a student's question","correct a student's error","give personalized learning support or emotional support","generate quiz questions or teaching material","grade a student's answer"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge","human"],"size":4000,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.16160","github":"https://github.com/ybai-nlp/EduBench","leaderboard":null},"good_for":["taxonomy","grading-design"],"fit":"Fits education assistants that serve both students and teachers. Borrow its nine-scenario taxonomy and the 12-indicator scoring sheet (scenario fit, factual accuracy, pedagogy), with human raters used to check the model grader.","caveats":"All prompts are LLM-synthesized, not real classroom data; scoring is LLM-judge on a 1 to 10 scale with modest human agreement (Kendall's W around 0.6); single-turn only."},{"id":"mathtutorbench","name":"MathTutorBench","year":2025,"by":"ETH Zurich (LRE lab) and TU Darmstadt","workloads":["education","knowledge-qa"],"task_types":["solve a math word problem","ask a Socratic question instead of giving the answer","judge whether a student's solution is correct and locate the mistake","write the next teacher turn that scaffolds the student","follow a stated pedagogy strategy over a dialog"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match","programmatic-check","llm-judge"],"size":null,"data":"mixed","pii":"none","license":"CC-BY-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2502.18940","github":"https://github.com/eth-lre/mathtutorbench","leaderboard":"https://eth-lre.github.io/mathtutorbench"},"good_for":["grading-design","taxonomy","baseline"],"fit":"Fits math tutoring agents that must teach rather than just answer. Borrow the three-skill split (math expertise, student understanding, pedagogical ability) and the small trained reward model that scores a teacher turn by win rate against the real teacher's turn.","caveats":"Dialog turns are scripted from existing datasets, not a live student; scaffolding scores depend on a 1.5B reward model trained by the authors; Socratic questioning is scored by BLEU; the README gives conflicting licenses (CC-BY-4.0 badge, CC-BY-SA-4.0 text)."},{"id":"tutorbench","name":"TutorBench","year":2025,"by":"Scale AI","workloads":["education","knowledge-qa"],"task_types":["explain a concept adapted to a specific student confusion","give actionable feedback on a student's work","write a hint that moves the student forward without solving it"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge"],"size":1490,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.02663","github":"https://huggingface.co/datasets/ScaleAI/TutorBench","leaderboard":"https://labs.scale.com/papers/tutorbench"},"good_for":["grading-design","case-template"],"fit":"Fits high-school and AP-level tutoring assistants. Borrow the per-sample rubric pattern, where each prompt ships with its own expert-written checklist that an LLM judge ticks off.","caveats":"Single-turn only; grading is entirely LLM-judge against rubrics; prompts and student states are expert-written, not real students; dataset is on Hugging Face but no runner code or license was found."},{"id":"clawsbench","name":"ClawsBench","year":2026,"by":"BenchFlow","workloads":["email-calendar","documents","workflow-automation","safety","general-assistant"],"task_types":["triage and reply to email","schedule or move calendar events","edit a Google Doc or move files in Drive","post or read messages in Slack","refuse or stop before an unsafe action (e.g. changing a contract silently)"],"interaction":"single-turn","environment":"api-sandbox","state_checked":true,"tools":["mock Gmail","Google Calendar","Docs","Drive and Slack APIs"],"grading":["programmatic-check","trajectory-check"],"size":44,"data":"synthetic","pii":"none","license":"CC-BY-NC-SA-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2604.05172","github":"https://github.com/benchflow-ai/ClawsBench","leaderboard":"https://clawsbench.benchflow.ai/"},"good_for":["sandbox-design","grading-design","taxonomy"],"fit":"Fits productivity assistants wired to Workspace and Slack. Borrow the mock services with snapshot/restore and the unsafe-action-rate metric reported alongside success.","caveats":"Only 44 tasks and the task definitions were still being released in batches; non-commercial license; grading method is thinly documented."},{"id":"workbench","name":"WorkBench","year":2024,"by":"Styles et al.","workloads":["email-calendar","crm","data-analysis","tool-calling","workflow-automation"],"task_types":["send an email or schedule a meeting","update a customer record","pull analytics numbers and report them","complete a project-management action","refuse or ask when a request is impossible in the sandbox"],"interaction":"single-turn","environment":"database","state_checked":true,"tools":["26 read/write tools over five sandbox databases"],"grading":["state-diff"],"size":690,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2405.00823","github":"https://github.com/olly-styles/WorkBench","leaderboard":null},"good_for":["grading-design","case-template","sandbox-design"],"fit":"Fits assistants that act on email, calendar and CRM data through tools. Borrow the outcome-centric grading that compares final database state regardless of action order and reports side effects separately.","caveats":"Small sandbox (a real inbox is far larger); single-turn only; 2024 baselines are stale."},{"id":"alfworld","name":"ALFWorld","year":2020,"by":"Shridhar et al. (Microsoft Research and University of Washington)","workloads":["embodied","planning"],"task_types":["find an object and put it somewhere in a simulated house","heat, cool or clean an object before placing it","examine an object under a light"],"interaction":"multi-turn-scripted","environment":"api-sandbox","state_checked":true,"tools":["text-world action interface"],"grading":["programmatic-check"],"size":null,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"saturated","links":{"paper":"https://arxiv.org/abs/2010.03768","github":"https://github.com/alfworld/alfworld","leaderboard":null},"good_for":["sandbox-design","baseline"],"fit":"Fits teams who need a cheap, deterministic multi-step environment for smoke tests of a planning loop; it is also the base of the household tasks in AgentBench, AgentBoard and AgentGym.","caveats":"From 2020 and reported near ceiling by many recent agent papers; a text game, not a work task; task counts were not visible on the README."},{"id":"apex-agents","name":"APEX-Agents","year":2026,"by":"Mercor","workloads":["finance","legal","documents","spreadsheets","general-assistant"],"task_types":["complete an investment banking analysis across spreadsheets and documents","draft or review corporate legal documents from a file set","build a consulting deliverable from mail, chat and files","find the relevant files in a messy simulated workspace"],"interaction":"single-turn","environment":"mixed","state_checked":true,"tools":["calendar","chat","code execution","documents","file system","mail","PDFs","spreadsheets","presentations"],"grading":["llm-judge"],"size":240,"data":"synthetic","pii":"none","license":"CC-BY-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2601.14242","github":"https://huggingface.co/datasets/mercor/apex-agents-v1.1","leaderboard":"https://mercor.com/apex/"},"good_for":["case-template","grading-design","sandbox-design"],"fit":"Fits agents doing analyst work in files and office tools. Borrow the \"world\" seeds (31 workspaces, 5,182 files) and the binary-criteria rubrics judged per criterion.","caveats":"Evaluation-only terms (no training); judge-graded; task count dropped from 480 to 240 between versions, so compare like with like."},{"id":"bizfinbench","name":"BizFinBench","year":2025,"by":"HiThink Research and Harbin Institute of Technology","workloads":["finance","knowledge-qa","data-analysis","tool-calling"],"task_types":["attribute an unusual market move to its cause","compute a financial number from given data","reason about dates and time windows in finance","describe a table of financial data in words","pick and fill the right financial tool call for a request","answer financial knowledge questions"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none; tool-use tasks are judged on the written call"],"grading":["llm-judge","exact-match"],"size":6781,"data":"mixed","pii":"none","license":"CC-BY-NC-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.19457","github":"https://github.com/HiThink-Research/BizFinBench","leaderboard":"https://hithink-research.github.io/BizFinBench.v2/"},"good_for":["grading-design","taxonomy","baseline"],"fit":"Fits retail-investor assistants and finance chat agents. Borrow the IteraJudge idea (an iterated judge that refines its decision boundary) and the nine-category task split when designing a judge for objective finance answers.","caveats":"Question answering only, not an environment; v1 is Chinese-only and v2 changed the question count, so quote the version; data is non-commercial (code is Apache-2.0); the judge is itself an LLM, so calibrate it before trusting scores."},{"id":"corpfin","name":"CorpFin (Vals AI)","year":2024,"by":"Vals AI","workloads":["finance","legal","documents","knowledge-qa"],"task_types":["extract a term or number from a long credit agreement","interpret a provision by cross-referencing earlier definitions","compute a numeric answer from agreement terms","judge whether a term is market standard"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge"],"size":858,"data":"real-public","pii":"none","license":"unknown","harness":["none"],"status":"saturated","links":{"paper":null,"github":null,"leaderboard":"https://www.vals.ai/benchmarks/corp_fin_v2","website":"https://www.vals.ai/benchmarks/corp_fin_v2"},"good_for":["baseline","taxonomy"],"fit":"Fits long-document review agents in corporate lending or legal operations. Borrow the question categories (extraction, interpretation, cross-reference, numeric reasoning, market standard) when writing cases over 200-page contracts.","caveats":"No paper or code; the 858-question test set is private and the page says v2 is archived because scores saturated; only a 20-question public validation set is available on request; document provenance is not stated, so treat the real-public label as an assumption."},{"id":"docmath-eval","name":"DocMath-Eval","year":2023,"by":"Yale NLP","workloads":["finance","documents","data-analysis"],"task_types":["answer a numeric question from a financial filing with text and tables","find the right table cells across a long document with many tables","do multi-step arithmetic and return a number","write a short program that computes the answer"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["programmatic-check"],"size":4000,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2311.09805","github":"https://github.com/yale-nlp/DocMath-Eval","leaderboard":null,"dataset":"https://huggingface.co/datasets/yale-nlp/DocMath-Eval"},"good_for":["grading-design","case-template","taxonomy"],"fit":"Fits finance or analyst agents that must compute figures from 10-K and 10-Q filings. Borrow its four-way split (simple vs complex reasoning, short vs long document) and its numeric answer checker that accepts chain-of-thought or program-of-thought output.","caveats":"Three of four subsets are re-annotated from older datasets (TAT-QA, FinQA, MultiHiertt, TAT-HQA), so overlap with those is built in; test-split answers were only released in 2026 and the official leaderboard is a CodaLab submission."},{"id":"failsafeqa","name":"FailSafeQA","year":2025,"by":"Writer","workloads":["finance","documents","safety"],"task_types":["answer a question over a long 10-K filing (4k to 27k tokens)","handle a misspelled, incomplete or off-domain version of the question","answer from OCR-degraded text","refuse when the provided document is missing or irrelevant"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge"],"size":220,"data":"mixed","pii":"none","license":"MIT","harness":["inspect_evals"],"status":"active","links":{"paper":"https://arxiv.org/abs/2502.06329","github":null,"leaderboard":null,"dataset":"https://huggingface.co/datasets/Writer/FailSafeQA"},"good_for":["grading-design","case-template","privacy-pattern"],"fit":"Fits document question answering agents in finance or any field where users upload the wrong file or type sloppy questions. Borrow its seven-condition design (clean query, three query perturbations, OCR noise, missing context, irrelevant context) and its robustness score that takes the minimum across conditions.","caveats":"Only 220 base items; grading relies on an LLM judge on a 1 to 6 scale; the queries are synthetic and contexts come only from SEC 10-K filings; the inspect_evals port is externally maintained at a pinned commit."},{"id":"finagentbench","name":"FinAgentBench","year":2025,"by":"LinqAlpha","workloads":["finance","documents","research"],"task_types":["choose which filing type (10-K, 10-Q, 8-K, earnings call) answers a question","rank passages within the chosen filing by relevance"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["programmatic-check"],"size":26000,"data":"real-public","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.14052","github":null,"leaderboard":null},"good_for":["taxonomy","grading-design"],"fit":"Fits retrieval-augmented filing agents where the retrieval step is the weak point. Borrow the two-stage split (document-type ranking then chunk ranking) and the ten query categories to diagnose where retrieval fails before judging final answers.","caveats":"Measures retrieval ranking (nDCG, MAP, MRR at 5) only, not end-to-end answers or tool use; no public data or code release was found at the time of cataloging; the size is given as about 26K in v4 while earlier versions reported 3,429."},{"id":"finance-agent-benchmark","name":"Finance Agent Benchmark (Vals AI)","year":2025,"by":"Vals AI","workloads":["finance","research","web-browsing","tool-calling","documents"],"task_types":["look up a figure or statement in a recent SEC filing","compare a metric across companies or periods","compute an adjusted or derived financial metric","judge whether results beat or missed expectations","build a simple projection or financial model","summarize market context for a company"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["web search (Tavily)","SEC EDGAR search","HTML page parser","recall of earlier results"],"grading":["llm-judge","exact-match"],"size":537,"data":"real-public","pii":"none","license":"CC-BY-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.00828","github":"https://github.com/vals-ai/finance-agent","leaderboard":"https://www.vals.ai/benchmarks/finance_agent"},"good_for":["grading-design","taxonomy","case-template","baseline"],"fit":"Fits research agents that answer analyst questions by searching the web and SEC filings. Borrow the nine-category task taxonomy and the per-fact rubric grading (separate correctness and contradiction checks per answer) rather than one overall judge call.","caveats":"Only 50 of 537 questions are public (CC-BY-4.0); 150 more are by request and the 337-question test set is private, so you can run the harness but not reproduce the leaderboard. Rubrics were drafted by GPT-4o. Live web search means answers can drift over time."},{"id":"financebench","name":"FinanceBench","year":2023,"by":"Patronus AI","workloads":["finance","documents","knowledge-qa"],"task_types":["answer a question about a public company from its 10-K, 10-Q, 8-K or earnings release","find the supporting evidence passage in the filing","compute a financial metric from reported numbers","refuse or flag when the filing does not contain the answer"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["retrieval over filing PDFs"],"grading":["human"],"size":150,"data":"real-public","pii":"none","license":"CC-BY-NC-4.0","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2311.11944","github":"https://github.com/patronus-ai/financebench","leaderboard":null,"dataset":"https://huggingface.co/datasets/PatronusAI/financebench"},"good_for":["case-template","baseline"],"fit":"Fits retrieval-augmented finance assistants that answer questions over company filings. Borrow its case format (question, answer, evidence string, justification, question type, reasoning type) as a template for a golden set over your own document corpus.","caveats":"Only 150 of 10,231 questions are public (the rest by request), grading in the paper was manual with no scoring script in the repo, and the license is non-commercial."},{"id":"finben","name":"FinBen","year":2024,"by":"The FinAI (Qianqian Xie et al.)","workloads":["finance","knowledge-qa","data-analysis"],"task_types":["classify sentiment of financial text","extract entities and numbers from filings","answer financial question-answering sets","summarize financial reports","score credit risk from tabular data","forecast stock movement from text"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match","programmatic-check"],"size":24,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2402.12659","github":"https://github.com/The-FinAI/PIXIU","leaderboard":"https://huggingface.co/spaces/TheFinAI/Open-FinLLM-Leaderboard"},"good_for":["baseline","taxonomy"],"fit":"Fits teams that need a broad baseline for a finance language model behind an agent rather than the agent itself. Borrow its grouping of 36 datasets into 24 task types to check which skills your model is missing.","caveats":"An aggregate of older public datasets, most single-turn and not agentic; the size of 24 counts task types, not questions; many component sets are saturated or leaked; licenses differ per dataset even though the code is MIT."},{"id":"fingaia","name":"FinGAIA","year":2025,"by":"SUFE AIFLM Lab (Shanghai University of Finance and Economics)","workloads":["finance","web-browsing","tool-calling","research","documents"],"task_types":["browse the web to find financial facts","read and process attached files (CSV, PDF, Word, audio, JSON)","run code or calculations on financial data","carry out multi-step analysis across securities, funds, banking or insurance"],"interaction":"single-turn","environment":"mixed","state_checked":false,"tools":["web browsing","file readers","code execution"],"grading":["exact-match"],"size":407,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2507.17186","github":"https://github.com/SUFE-AIFLM-Lab/FinGAIA","leaderboard":null},"good_for":["case-template","taxonomy","baseline"],"fit":"Fits general finance research agents in the GAIA style (web, files, code). Borrow the three scenario depths (operational analytics, asset decision, strategic risk) and the capability breakdown per task to size a case set.","caveats":"Chinese-language tasks about the Chinese financial market; only part of the 407 tasks is released; the scoring procedure beyond accuracy is not described; web-dependent answers can drift."},{"id":"finqa","name":"FinQA","year":2021,"by":"UC Santa Barbara and J.P. Morgan","workloads":["finance","documents","data-analysis"],"task_types":["answer a numeric question from an earnings report table and text","produce the step-by-step reasoning program that yields the answer","pick the right numbers out of a table and the surrounding paragraphs"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match","programmatic-check"],"size":8281,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2109.00122","github":"https://github.com/czyssrs/FinQA","leaderboard":"https://codalab.lisn.upsaclay.fr/competitions/1846"},"good_for":["grading-design","case-template","baseline"],"fit":"Fits finance agents that compute figures from reports and must show their work. Borrow its two-level grading (execution accuracy on the final number plus program accuracy on the reasoning steps) when you want to check reasoning, not just the answer.","caveats":"From 2021 and widely trained on since, so leakage into current models is likely; questions cover S&P 500 earnings reports only; a private test set exists but needs a CodaLab submission."},{"id":"finsearchcomp","name":"FinSearchComp","year":2025,"by":"ByteDance Seed","workloads":["finance","research","web-browsing","tool-calling"],"task_types":["fetch a current market figure from a live source","look up a historical financial fact for a named company or index","run a multi-step investigation that combines several figures with calculation"],"interaction":"single-turn","environment":"mixed","state_checked":false,"tools":["web search","financial data APIs or plugins"],"grading":["llm-judge"],"size":635,"data":"real-public","pii":"none","license":"CC-BY-4.0","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2509.13160","github":"https://huggingface.co/datasets/ByteSeedXpert/FinSearchComp","leaderboard":"https://xpert.bytedance.com/leaderboard?id=FinSearchComp"},"good_for":["grading-design","case-template","baseline"],"fit":"Fits finance research or market-data agents that must search and then compute. Borrow the three-tier task split (time-sensitive fetch, simple lookup, complex investigation) and the judge prompts with explicit numeric tolerance bands shipped with each question.","caveats":"The time-sensitive subset has answers that change daily and needs a live data feed to grade; judge prompts are written in Chinese; half the set is Greater China markets; only data and judge prompts are released, no agent runner; the arXiv page lists CC-BY-NC-SA-4.0 while the dataset card says CC-BY-4.0."},{"id":"tat-qa","name":"TAT-QA","year":2021,"by":"NExT++ (National University of Singapore)","workloads":["finance","documents","data-analysis"],"task_types":["answer a question from a table plus the paragraphs around it in a financial report","add, subtract, count or compare numbers drawn from a table and text","return the right scale and unit with a numeric answer"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match"],"size":16552,"data":"real-public","pii":"none","license":"CC-BY-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2105.07624","github":"https://github.com/NExTplusplus/TAT-QA","leaderboard":null},"good_for":["case-template","grading-design","baseline"],"fit":"Fits agents that read financial statements where the answer needs both a table and the surrounding text. Borrow its answer-type tagging (span, multi-span, count, arithmetic) and its exact-match and F1 scorer with scale handling.","caveats":"From 2021 and pre-dates current models; DocMath-Eval re-annotated part of it, which suggests label noise; the 2,757 contexts come from a limited set of report layouts."},{"id":"underwrite","name":"UNDERWRITE","year":2026,"by":"Snorkel AI","workloads":["finance","customer-support","tool-calling"],"task_types":["gather applicant details from a simulated user over several turns","look up underwriting guidelines through noisy tools","decide whether and how to underwrite an insurance policy"],"interaction":"multi-turn-simulated-user","environment":"api-sandbox","state_checked":false,"tools":["underwriting knowledge and policy lookup tools"],"grading":["programmatic-check","pass^k"],"size":300,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2602.00456","github":null,"leaderboard":"https://snorkel.ai/leaderboard/snorkel-underwrite/"},"good_for":["user-simulator","grading-design"],"fit":"Fits insurance and lending agents that must collect information before deciding. Borrow the imperfect simulated user and the pass^k reliability drop as a reported metric.","caveats":"Vendor benchmark with no public code or task data; leaderboard already shows top models above 90%."},{"id":"balrog","name":"BALROG","year":2024,"by":"Paglieri et al.","workloads":["games","planning"],"task_types":["play a roguelike or grid-world game from text or image observations","explore, gather and survive over long episodes","follow instructions in a text-based game"],"interaction":"autonomous-long-horizon","environment":"api-sandbox","state_checked":false,"tools":["game action interface"],"grading":["programmatic-check"],"size":null,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2411.13543","github":null,"leaderboard":"https://balrogai.com"},"good_for":["grading-design","sandbox-design"],"fit":"Fits teams studying long-horizon decision making with partial credit; borrow the per-game progress metrics that give signal even when a run never finishes.","caveats":"Games only, no work tasks; the repo link was not on the paper page (code is pointed to via balrogai.com) and the license was not seen."},{"id":"agencybench","name":"AgencyBench","year":2026,"by":"GAIR (Shanghai Jiao Tong University)","workloads":["general-assistant","coding","research","tool-calling"],"task_types":["build a working front-end or game from a brief, then revise it on feedback","implement back-end or code tasks in a sandbox","research a topic and deliver a written result","use MCP tools across a long session (about 90 tool calls)"],"interaction":"multi-turn-simulated-user","environment":"containers","state_checked":false,"tools":["Docker sandbox","browser","MCP tools"],"grading":["llm-judge","programmatic-check"],"size":138,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2601.11044","github":"https://github.com/GAIR-NLP/AgencyBench","leaderboard":null},"good_for":["user-simulator","grading-design","baseline"],"fit":"Fits long-running coding or deliverable-building agents that take iterative feedback. Borrow the user-simulation agent that supplies feedback and the mixed rule, vision and LLM judges on deliverables.","caveats":"Runs cost about a million tokens per scenario; rubric scoring is largely LLM-judged; results depend strongly on the scaffold used."},{"id":"agentatlas","name":"AgentAtlas","year":2026,"by":"Mazaheri and Mazaheri","workloads":["general-assistant","safety","planning"],"task_types":["classify each agent step as act, ask, refuse, stop, confirm or recover","label a failed trajectory by error source and downstream impact","audit a benchmark for which behaviors it actually measures"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["trajectory-check"],"size":null,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2605.20530","github":null,"leaderboard":null},"good_for":["taxonomy"],"fit":"Fits any team deciding what to measure beyond pass/fail: the six control decisions and nine failure categories are a ready vocabulary for labeling an agent's trajectories.","caveats":"A taxonomy and audit paper, not a runnable benchmark; the 1,342-item synthetic study set is explicitly not released, and no code repository was found."},{"id":"agentbench","name":"AgentBench","year":2023,"by":"Tsinghua University (THUDM)","workloads":["general-assistant","it-ops","sql","shopping","games"],"task_types":["run shell commands to complete an operating system task","write and run SQL to answer or change a database","query a knowledge graph with tools","play a digital card game against an opponent","complete household tasks in a text world","shop or browse on simulated web pages"],"interaction":"multi-turn-scripted","environment":"mixed","state_checked":true,"tools":["shell","SQL","knowledge-graph API","game and web simulators"],"grading":["programmatic-check","state-diff"],"size":null,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2308.03688","github":"https://github.com/THUDM/AgentBench","leaderboard":null},"good_for":["sandbox-design","baseline","taxonomy"],"fit":"Fits teams who want one harness covering shell, database and web-style tasks to sanity-check a general agent. Borrow the containerized per-environment servers; the 2025 function-calling version keeps it usable with current tool-call APIs.","caveats":"Several sub-environments wrap older benchmarks (ALFWorld, WebShop, Mind2Web) that are near ceiling for recent models; per-environment task counts are only in an image in the README."},{"id":"agentboard","name":"AgentBoard","year":2024,"by":"Ma et al. (HKUST and others)","workloads":["general-assistant","embodied","games","web-browsing","tool-calling"],"task_types":["complete household or science tasks in a text world","solve a text adventure or PDDL planning problem","shop or navigate on simulated websites","answer questions by calling tools or operate a tool to change state"],"interaction":"multi-turn-scripted","environment":"mixed","state_checked":true,"tools":["text-world simulators","web simulators","query and operation tools"],"grading":["programmatic-check"],"size":100,"data":"mixed","pii":"none","license":"Apache-2.0 (code); GPL-2.0 (data)","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2401.13178","github":"https://github.com/hkust-nlp/AgentBoard","leaderboard":"https://hkust-nlp.github.io/agentboard/static/leaderboard.html"},"good_for":["grading-design","taxonomy"],"fit":"Fits teams who want partial credit instead of pass/fail for multi-step tasks. Borrow the progress-rate metric (sub-goals reached) and the grounding-accuracy breakdown.","caveats":"Nine environments all wrap earlier benchmarks, so much of it is familiar to current models; the dataset is GPL-2.0, which some teams cannot redistribute."},{"id":"agentgym","name":"AgentGym","year":2024,"by":"Xi et al.","workloads":["general-assistant","embodied","web-browsing","tool-calling","games"],"task_types":["shop on a simulated store or act on simulated websites","complete household, science or crafting tasks in text worlds","solve mazes and word games","call weather, movie, academic, spreadsheet and to-do tools","write SQL for a database question"],"interaction":"multi-turn-scripted","environment":"mixed","state_checked":true,"tools":["per-environment HTTP servers"],"grading":["programmatic-check"],"size":null,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2406.04151","github":"https://github.com/WooooDyy/AgentGym","leaderboard":null},"good_for":["sandbox-design","baseline"],"fit":"Fits teams who want many environments behind one HTTP interface for both training and evaluation of a general agent. Borrow the server-per-environment layout and the AgentEval held-out split.","caveats":"Fourteen environments wrap existing benchmarks and were used to train the authors' models, so held-out status is only as good as the split; evaluation item counts are summed from a table, not stated."},{"id":"agentrx","name":"AgentRx","year":2026,"by":"Microsoft Research","workloads":["general-assistant","it-ops","tool-calling","web-browsing"],"task_types":["locate the critical failing step in a failed agent run","assign the failure to a category from a cross-domain taxonomy","check a trajectory against synthesized constraints and log violations"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match","llm-judge"],"size":170,"data":"real-public","pii":"none","license":"CC-BY-4.0","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2602.02475","github":null,"leaderboard":null},"good_for":["taxonomy","grading-design"],"fit":"Fits teams debugging a deployed agent across API workflows, incident management or web tasks. Borrow the constraint-synthesis-then-check pattern that gives an auditable violation log before the judge decides.","caveats":"Only 170 failed runs over 11 settings; the dataset on Hugging Face is gated; no public code link was seen on the paper page."},{"id":"appworld","name":"AppWorld","year":2024,"by":"Stony Brook University / AI2","workloads":["general-assistant","tool-calling","email-calendar","shopping","planning"],"task_types":["complete a day-to-day errand across several apps (shopping, music, payments, messaging)","write code that calls app APIs with loops and conditions","read data from one app and act on it in another","avoid collateral changes while completing the task"],"interaction":"single-turn","environment":"api-sandbox","state_checked":true,"tools":["457 REST APIs over 9 consumer apps","Python code execution"],"grading":["state-diff","unit-tests"],"size":750,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["inspect_evals","hal","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2407.18901","github":"https://github.com/stonybrooknlp/appworld","leaderboard":"https://appworld.dev/appworld/leaderboard"},"good_for":["grading-design","sandbox-design","case-template","baseline"],"situations":["unanswerable"],"fit":"Fits assistants that drive several apps through APIs on a user's behalf. Borrow its database-state unit tests that accept any valid path and also check for unwanted side effects.","caveats":"Consumer apps only (no enterprise systems); test-set solutions are shipped encrypted, and the agent must write Python rather than call tools."},{"id":"functionchat-bench","name":"FunctionChat-Bench","year":2024,"by":"Kakao","workloads":["general-assistant","tool-calling"],"task_types":["produce a correct tool call from a user turn","complete the answer in natural language after a tool result","ask a slot question when required arguments are missing","detect that no available tool is relevant and say so"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["25 functions across 5 tool-set types, Korean dialogues"],"grading":["llm-judge"],"size":700,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2411.14054","github":"https://github.com/kakao/FunctionChat-Bench","leaderboard":null},"good_for":["taxonomy","grading-design"],"fit":"Fits chat assistants where every turn must be one of call, answer, ask, or refuse. The four-way output taxonomy and the rubric-based pass/fail judge are a compact grading design to reuse.","caveats":"Korean only; the dialog set is 200 items from 45 conversations, and scoring depends on a GPT-4 judge."},{"id":"gaia","name":"GAIA","year":2023,"by":"Meta FAIR / Hugging Face","workloads":["general-assistant","research","web-browsing","knowledge-qa"],"task_types":["answer a question that needs web browsing and several reasoning steps","read an attached file (spreadsheet, PDF, image, audio) to find the answer","combine facts from several sources into one short answer"],"interaction":"single-turn","environment":"mixed","state_checked":false,"tools":["web search","browser","file readers","code execution"],"grading":["exact-match"],"size":466,"data":"real-public","pii":"none","license":"unknown","harness":["inspect_evals","hal","own-repo"],"status":"saturated","links":{"paper":"https://arxiv.org/abs/2311.12983","github":"https://huggingface.co/datasets/gaia-benchmark/GAIA","leaderboard":"https://huggingface.co/spaces/gaia-benchmark/leaderboard"},"good_for":["grading-design","baseline","case-template"],"fit":"Fits research-style assistants that must return one checkable answer. Borrow the short-answer exact-match format and the three difficulty levels.","caveats":"Read-only question answering, no state changes; top agents now score high; answers to 300 test questions are private and the dataset is gated."},{"id":"gaia2","name":"Gaia2 (Meta ARE)","year":2025,"by":"Meta Superintelligence Labs","workloads":["general-assistant","email-calendar","tool-calling","multi-agent","planning"],"task_types":["handle a personal-assistant request across messaging, contacts, calendar and email apps","act at the right time when an event is scheduled","ask for clarification when the request is ambiguous","adapt when the environment changes mid-task or returns noisy results","coordinate with another agent to finish a task"],"interaction":"multi-turn-simulated-user","environment":"api-sandbox","state_checked":true,"tools":["smartphone-style apps in the ARE simulator"],"grading":["trajectory-check","llm-judge"],"size":800,"data":"synthetic","pii":"none","license":"CC-BY-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2602.11964","github":"https://github.com/facebookresearch/meta-agents-research-environments","leaderboard":"https://huggingface.co/datasets/meta-agents-research-environments/gaia2"},"good_for":["sandbox-design","grading-design","taxonomy","user-simulator"],"fit":"Fits personal and workplace assistants that must act on time and cope with events arriving during a task. Borrow the write-action verifier against oracle events and the capability splits (execution, search, ambiguity, adaptability, time, noise, agent-to-agent).","caveats":"Consumer apps, not enterprise systems; synthetic Llama-generated content with Llama license strings attached; scenario counts differ between the paper and dataset card."},{"id":"hle","name":"Humanity's Last Exam","year":2025,"by":"Center for AI Safety and Scale AI","workloads":["general-assistant","knowledge-qa"],"task_types":["answer an expert-level closed-book question in math, science or humanities","pick the right option on a hard multiple-choice question","state a confidence that matches accuracy"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge","exact-match"],"size":2500,"data":"real-public","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2501.14249","github":"https://github.com/centerforaisafety/hle","leaderboard":"https://lastexam.ai"},"good_for":["baseline","grading-design"],"fit":"Fits general assistants where raw expert knowledge of the underlying model matters. Borrow the short-answer judge prompt and the calibration-error reporting next to accuracy.","caveats":"Not an agent benchmark: closed-book expert questions with no tools or multi-step work; some questions have multimodal inputs; scores say little about tool use or task completion."},{"id":"lost-in-conversation","name":"Lost in Conversation","year":2025,"by":"Microsoft Research","workloads":["general-assistant","coding","sql","tool-calling","documents"],"task_types":["complete a task whose requirements arrive one piece per turn","write code or SQL after several underspecified turns","produce an API call once enough details are revealed","avoid committing to an early answer and refusing to revise it"],"interaction":"multi-turn-simulated-user","environment":"static","state_checked":false,"tools":["none; tasks graded by task-specific tests or references"],"grading":["unit-tests","exact-match","llm-judge"],"size":600,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.06120","github":"https://github.com/microsoft/lost_in_conversation","leaderboard":null},"good_for":["user-simulator","grading-design"],"fit":"Fits any chat agent whose users reveal requirements gradually. Borrow the sharded instruction format, the simulator that reveals one shard per turn, and the split of results into aptitude (90th percentile) and unreliability (90th minus 10th).","caveats":"Not a leaderboard benchmark but a simulation study; the 600 instructions are derived from existing public datasets and the simulated user is GPT-4o-mini."},{"id":"metatool","name":"MetaTool","year":2023,"by":"Lehigh University","workloads":["general-assistant","tool-calling"],"task_types":["decide whether a request needs a tool at all","choose one tool among several with similar descriptions","choose tools for a request that needs more than one","avoid an unreliable tool when a safer one exists"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["195 tool descriptions derived from the OpenAI plugin list"],"grading":["exact-match"],"size":21127,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2310.03128","github":"https://github.com/HowieHwong/MetaTool","leaderboard":null},"good_for":["taxonomy","case-template"],"fit":"Fits any agent with a large tool catalog where the first failure is picking the wrong tool. Its four selection subtasks (similar, scenario-specific, reliability, multi-tool) are a ready taxonomy for tool-routing test cases.","caveats":"Selection only, no execution and no conversation; queries are ChatGPT-generated and the repo has been idle since early 2024."},{"id":"occubench","name":"OccuBench","year":2026,"by":"Xiaomeng Hu et al. (GregxmHu on GitHub)","workloads":["general-assistant","tool-calling","workflow-automation"],"task_types":["complete a job-role task by calling domain tools","keep going when a tool returns an explicit error","detect silently degraded or wrong tool data","finish a multi-step professional scenario"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["LLM-simulated domain tools"],"grading":["llm-judge"],"size":382,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2604.10866","github":"https://github.com/GregxmHu/OccuBench","leaderboard":null},"good_for":["sandbox-design","grading-design"],"fit":"Fits tool-using agents in niche occupations where no real sandbox exists. Borrow its idea of an LLM that plays the environment, and its fault-injection modes (explicit error, silent data degradation, mixed).","caveats":"Environment and tool responses are themselves generated by a model, so results depend on simulator quality; 100 scenarios across 65 domains is thin per domain; not built from O*NET; rubric judge is a majority vote of models, not humans; no data license stated."},{"id":"remote-labor-index","name":"Remote Labor Index (RLI)","year":2025,"by":"Scale AI / Center for AI Safety","workloads":["general-assistant","documents","data-analysis","coding"],"task_types":["deliver a complete freelance project from a brief and input files","produce media, CAD, web, design or data deliverables","draft legal or business documents to a client's spec"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["any; deliverables are judged","not the process"],"grading":["human"],"size":240,"data":"real-public","pii":"possible","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.26787","github":null,"leaderboard":"https://scale.com/leaderboard/rli"},"good_for":["baseline","grading-design","taxonomy"],"fit":"Fits teams who want an economic value baseline for end-to-end work. Borrow the 3-point expert rating with majority vote and the Bradley-Terry pairing against a human reference.","caveats":"Only 10 of 240 projects are public; human grading only; projects skew to media and design rather than office workflows."},{"id":"theagentcompany","name":"TheAgentCompany","year":2024,"by":"Carnegie Mellon University","workloads":["general-assistant","coding","hr","documents","workflow-automation"],"task_types":["complete a work task using GitLab, a file share, a chat tool and a project tracker","ask a simulated colleague for missing information over chat","screen resumes or fill HR and admin spreadsheets","write or review code and open issues","produce a document or report and save it to the shared drive"],"interaction":"multi-turn-simulated-user","environment":"containers","state_checked":true,"tools":["browser","terminal","GitLab","ownCloud","RocketChat","Plane"],"grading":["programmatic-check","llm-judge"],"size":175,"data":"synthetic","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2412.14161","github":"https://github.com/TheAgentCompany/TheAgentCompany","leaderboard":"https://the-agent-company.com/#/leaderboard"},"good_for":["sandbox-design","user-simulator","grading-design","baseline"],"situations":["ambiguous","multi-turn"],"fit":"Fits digital-worker agents that touch several internal tools and must ask coworkers for details. Borrow the self-hosted company sandbox, the LLM-played colleagues and the checkpoint-plus-result scoring.","caveats":"Skewed toward software engineering tasks; heavy Docker setup; some checkpoints are LLM-judged."},{"id":"tooldial","name":"ToolDial","year":2025,"by":"Seoul National University","workloads":["general-assistant","tool-calling","knowledge-qa"],"task_types":["ask a clarifying question when an API parameter is missing","chain a second API call to obtain a required input","track the dialogue state (API, parameters, values) each turn","choose the next system action from 16 dialogue acts"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["473 RapidAPI APIs linked by an input-output compatibility graph"],"grading":["exact-match","llm-judge"],"size":1166,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2503.00564","github":"https://github.com/holi-lab/ToolDial","leaderboard":null},"good_for":["case-template","taxonomy","user-simulator"],"fit":"Fits assistants that must ask for missing information before calling an API. The action vocabulary (request, clarify, fail inform) and the API-graph trick for generating dependent calls are worth copying into a case generator.","caveats":"Dialogues are fully GPT-4o generated and scripted, so the model is graded against one expected path; 11,111 dialogues in total but only 1,166 are test."},{"id":"tooleyes","name":"ToolEyes","year":2024,"by":"Fudan University","workloads":["general-assistant","tool-calling","finance","research"],"task_types":["understand the intent behind a real-world request","plan which tools to call and in what order","call a real tool with correct format and parameters","organize the tool output into a usable answer"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["568 real tools in 7 scenarios (text generation, data understanding, real-time search, app manipulation, personal life, information retrieval, financial transactions)"],"grading":["llm-judge","programmatic-check"],"size":382,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2401.00741","github":"https://github.com/Junjie-Ye/ToolEyes","leaderboard":null},"good_for":["grading-design","taxonomy"],"fit":"Fits general assistants with a broad tool library. Borrow the five-dimension scoring (format, intent, planning, tool selection, answer) that mixes rule checks with a judge instead of grading only the final answer.","caveats":"Depends on live third-party tools and GPT-4 judging, so results drift; run scripts are not documented in the README."},{"id":"tooltalk","name":"ToolTalk","year":2023,"by":"Microsoft","workloads":["general-assistant","email-calendar","tool-calling"],"task_types":["set an alarm or reminder from a chat request","send an email or message on the user's behalf","add or change a calendar event over several turns","look up the weather or an account detail","carry a multi-step request across turns without redundant actions"],"interaction":"multi-turn-scripted","environment":"api-sandbox","state_checked":false,"tools":["28 simulated personal-assistant tools in 7 plugins (account, alarm, calendar, email, message, reminder, weather)"],"grading":["exact-match","trajectory-check"],"size":78,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2311.10775","github":"https://github.com/microsoft/ToolTalk","leaderboard":null},"good_for":["case-template","grading-design","sandbox-design"],"fit":"Fits personal-assistant agents that act on calendars, email and messages in a conversation. Borrow the simulated tool implementations with mock databases and the strict success rule (perfect recall of required calls, zero wrong actions).","caveats":"Only 78 handwritten conversations, the user side is scripted, and the repo has not moved since 2024; results from 2023 models are saturated by current ones."},{"id":"taxcalcbench","name":"TaxCalcBench","year":2025,"by":"Column Tax","workloads":["government","finance","documents"],"task_types":["compute a US federal and state income tax return from taxpayer data","read W-2 and 1099 PDFs and fill Form 1040 lines","apply filing-status and credit eligibility rules","use tax tables without arithmetic errors"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["optional web search"],"grading":["exact-match","pass^k"],"size":50,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2507.16126","github":"https://github.com/column-tax/tax-calc-bench","leaderboard":null},"good_for":["grading-design","case-template","privacy-pattern"],"fit":"Fits tax-preparation, benefits-eligibility or any form-filling agent whose output is a set of numbered lines. Borrow its line-by-line strict and lenient (plus or minus 5 dollars) scoring, pass^k over repeat runs, and fully synthetic taxpayer cases.","caveats":"Tiny (50 TY25 cases plus 51 legacy TY24 cases); US personal income tax only; cases are simplified versions of the vendor's own engine tests; models solved under a third of returns, so it mostly measures failure."},{"id":"agentclinic","name":"AgentClinic","year":2024,"by":"Schmidgall et al. (Johns Hopkins, Stanford, ETH Zurich)","workloads":["healthcare","multi-agent","knowledge-qa"],"task_types":["interview a simulated patient to collect history and symptoms","order exams and imaging from a measurement agent","reach a diagnosis under incomplete information within a turn budget","use tools such as a persistent notebook or retrieval across cases"],"interaction":"multi-turn-simulated-user","environment":"static","state_checked":false,"tools":["LLM patient agent","LLM measurement agent","notebook","web and textbook retrieval"],"grading":["llm-judge"],"size":null,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2405.07960","github":"https://github.com/SamuelSchmidgall/AgentClinic","leaderboard":"https://agentclinic.github.io/"},"good_for":["user-simulator","case-template"],"fit":"Fits diagnostic or triage assistants that must gather information by asking questions rather than reading a full vignette. Borrow the four-agent setup (patient, doctor, measurement, moderator), the bias injection options, and the structured case JSON.","caveats":"Cases are MedQA exam vignettes (215) and NEJM case challenges (120) rewritten by GPT-4 into patient scripts, so the base sets are synthetic; the optional AgentClinic-MIMIC-IV suite uses real records and needs PhysioNet credentialed access; the moderator is an LLM judge of diagnosis only; the repo has not been updated since September 2024."},{"id":"clibench","name":"CliBench","year":2024,"by":"Ma et al. (UCLA)","workloads":["healthcare","knowledge-qa"],"task_types":["list discharge diagnoses as ICD-10-CM codes from an admission record","identify procedures performed as ICD-10-PCS codes","order the lab tests a clinician would order (LOINC)","prescribe medications as ATC codes"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["programmatic-check"],"size":null,"data":"real-public","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2406.09923","github":"https://github.com/CliBench/CliBench","leaderboard":"https://clibench.github.io/"},"good_for":["grading-design","taxonomy"],"fit":"Fits coding, order-set or discharge-summary assistants that output sets of standard codes. Borrow the multi-granular scoring that maps predicted codes to ontology ancestors and reports precision, recall and F1 at each level.","caveats":"Zero-shot generation from a record, not agentic; the evaluation set (about 1,000 admissions per task) must be rebuilt from raw PhysioNet credentialed MIMIC-IV, nothing ships; the data license means models trained on it should be treated as containing sensitive data."},{"id":"clinicalbench","name":"ClinicalBench (Can LLMs Beat Traditional ML Models in Clinical Prediction?)","year":2024,"by":"Chen, Yu and Shu et al. (Emory, Northwestern, Harvard, Cornell)","workloads":["healthcare","data-analysis"],"task_types":["predict length of stay from an admission record","predict in-hospital mortality","predict readmission within two weeks"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["programmatic-check"],"size":null,"data":"real-public","pii":"possible","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2411.06469","github":"https://github.com/canyuchen/ClinicalBench","leaderboard":null},"good_for":["baseline","grading-design"],"fit":"Fits teams tempted to use an LLM for risk prediction over structured records. Borrow the matched-cohort comparison against 11 traditional ML baselines and the rule that unparseable answers count as wrong.","caveats":"This is the Chen et al. KDD 2026 ClinicalBench, the only benchmark by that name we could verify; prediction from prompts, not agentic; prompts must be rebuilt from PhysioNet credentialed MIMIC-III and MIMIC-IV; headline finding is that XGBoost and SVM beat every LLM, so it is a sanity check rather than a target."},{"id":"ehragent","name":"EHRAgent (EHRSQL-derived agent evaluation)","year":2024,"by":"Shi, Xu et al. (Georgia Tech, Emory)","workloads":["healthcare","sql","data-analysis","tool-calling"],"task_types":["answer a clinician's question over multi-table EHR data by writing and running code","find and join records across patient, admission, lab and prescription tables","compute aggregates such as counts, averages or survival rate","repair generated code from execution error messages"],"interaction":"autonomous-long-horizon","environment":"database","state_checked":false,"tools":["Python code executor","table lookup and SQL helper tools","medical knowledge lookup"],"grading":["exact-match"],"size":2156,"data":"real-public","pii":"possible","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2401.07128","github":"https://github.com/wshi83/EhrAgent","leaderboard":null},"good_for":["case-template","baseline"],"fit":"Fits data-retrieval agents that let clinicians ask questions of structured EHR tables in plain language. Borrow the code-as-plan action format, the interactive debugging loop, and the question complexity levels (number of tables joined).","caveats":"A method paper whose evaluation set is a re-processed slice of EHRSQL (580 MIMIC-III, 580 eICU, 996 TREQS questions) distributed through a Google Drive link; the underlying MIMIC-III and eICU databases need PhysioNet credentialed access; no license file in the repo; success rate is a single exact-match answer check."},{"id":"ehrnoteqa","name":"EHRNoteQA","year":2024,"by":"Kweon, Kim et al. (KAIST, Edward Choi's group)","workloads":["healthcare","documents","knowledge-qa"],"task_types":["answer a clinician's question from one or more discharge summaries","find information spread across several admissions of the same patient","choose the answer in multiple-choice form"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge","exact-match"],"size":962,"data":"real-public","pii":"possible","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2402.16040","github":"https://github.com/ji-youn-kim/EHRNoteQA","leaderboard":null},"good_for":["case-template","grading-design"],"fit":"Fits chart-summarization or note question-answering assistants that read long discharge summaries. Borrow the clinician-reviewed question generation over multiple notes and the reported correlation with clinician judgments as a way to validate a proxy benchmark.","caveats":"Question answering only, no tools; the dataset and the MIMIC-IV-Note text both sit behind PhysioNet credentialed access and a HIPAA-compliant model endpoint is required; open-ended scoring uses a GPT-4 judge; MIT covers the code, the data carries the PhysioNet license."},{"id":"ehrsql","name":"EHRSQL","year":2022,"by":"Lee et al. (KAIST, Edward Choi's group)","workloads":["healthcare","sql"],"task_types":["translate a hospital staff question into SQL over MIMIC-III or eICU","resolve relative time expressions such as last year or since admission","abstain when a question cannot be answered from the database"],"interaction":"single-turn","environment":"database","state_checked":false,"tools":["SQLite database"],"grading":["exact-match","programmatic-check"],"size":24411,"data":"real-public","pii":"possible","license":"CC-BY-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2301.07695","github":"https://github.com/glee4810/EHRSQL","leaderboard":null},"good_for":["case-template","grading-design","privacy-pattern"],"fit":"Fits text-to-SQL or data-retrieval agents over hospital databases. Borrow the questions collected from 222 real hospital staff, the time-expression templates, the unanswerable question split, and the execution-based F1 that penalizes confident wrong answers.","caveats":"Single-shot text-to-SQL, no agent loop; shipped SQLite databases have shuffled patient values and rebuilding from source needs PhysioNet credentialed MIMIC-III and eICU; the MIMIC-IV version (EHRSQL-2024 shared task) lives in a separate repo; annotations were revised in 2026 (v1.5) so older numbers are not comparable."},{"id":"fhir-agentbench","name":"FHIR-AgentBench","year":2025,"by":"Verily, KAIST and MIT (Lee, Pollard, Johnson, Choi)","workloads":["healthcare","tool-calling","knowledge-qa"],"task_types":["answer a clinician's question by querying a FHIR server","pick the right FHIR resource types and search parameters","reason over retrieved resources in natural language or by running code","return no result when no record matches"],"interaction":"autonomous-long-horizon","environment":"api-sandbox","state_checked":false,"tools":["FHIR REST API on a Google Cloud Healthcare store","Python code interpreter"],"grading":["exact-match","programmatic-check"],"size":2931,"data":"real-public","pii":"possible","license":"CC-BY-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2509.19319","github":"https://github.com/glee4810/FHIR-AgentBench","leaderboard":null},"good_for":["case-template","grading-design","sandbox-design"],"fit":"Fits read-only EHR question-answering agents that talk to a FHIR server. Borrow the two-stage scoring (resource retrieval recall and precision, then answer correctness) and the single-turn versus multi-turn and text versus code agent variants.","caveats":"Read-only, no orders or writes; requires setting up a Google Cloud FHIR store; built on the 100-patient MIMIC-IV-FHIR demo (PhysioNet, ODbL) so the record space is small; questions are reused from EHRSQL-2024; the best agent reaches only 50 percent."},{"id":"healthbench","name":"HealthBench","year":2025,"by":"OpenAI","workloads":["healthcare","knowledge-qa","safety"],"task_types":["answer a health question from a member of the public","handle a clinician request such as summarizing notes or drafting a message","recognize an emergency and refer the user","ask for missing context before answering","tailor depth and wording to the user's expertise"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge"],"size":5000,"data":"mixed","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.08775","github":"https://github.com/openai/simple-evals","leaderboard":null},"good_for":["grading-design","case-template","taxonomy"],"fit":"Fits patient-facing or clinician-facing chat assistants with no tools. Borrow the per-conversation weighted rubric criteria (48,562 written by 262 physicians), the seven themes and behavior axes, and the Consensus and Hard subsets.","caveats":"Conversation-only, no tool use or state; most conversations were synthetically generated from physician-written situations; grading depends on a model grader (GPT-4.1 in the reference code); authored by a model vendor; simple-evals is otherwise deprecated and only keeps HealthBench as a reference implementation."},{"id":"medagentbench","name":"MedAgentBench","year":2025,"by":"Stanford ML Group and Stanford Medicine (Jiang, Ng, Chen)","workloads":["healthcare","tool-calling","workflow-automation"],"task_types":["look up a patient's lab results, vitals or medications through FHIR queries","compute a derived value from a chart such as an average or a trend","place an order or record an observation by POSTing to the FHIR server","carry out a short clinician-written instruction over one patient's record"],"interaction":"autonomous-long-horizon","environment":"api-sandbox","state_checked":false,"tools":["FHIR REST API (GET and POST) on a HAPI FHIR server in Docker"],"grading":["programmatic-check"],"size":300,"data":"real-public","pii":"possible","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2501.14654","github":"https://github.com/stanfordmlgroup/MedAgentBench","leaderboard":null},"good_for":["sandbox-design","case-template","grading-design"],"fit":"Fits agents that read and write an electronic health record through standard APIs (chart review, order entry, documentation helpers). Borrow the Docker FHIR sandbox, the 10 task categories written by physicians, and the rule-based checks on POST payloads.","caveats":"Patient profiles are real Stanford (STARR) records that were de-identified and time-jittered, not synthetic; grading checks the request payload rather than diffing server state; the reference solutions file must be fetched separately from a Stanford Box link; only 300 tasks and the best model already scores about 70 percent."},{"id":"medagentgym","name":"MedAgentGym","year":2025,"by":"Xu, Shi et al. (Emory, UT Southwestern, Georgia Tech)","workloads":["healthcare","coding","data-analysis","sql"],"task_types":["write and run code to answer a question over EHR tables","perform a clinical calculation in code","solve a bioinformatics or biomedical data-science coding task","train or run a prediction model on clinical data","debug from execution feedback inside a sandbox"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":false,"tools":["Python sandbox","SQL","terminal","debug action"],"grading":["exact-match","programmatic-check"],"size":72413,"data":"mixed","pii":"possible","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.04405","github":"https://github.com/wshi83/MedAgentGym","leaderboard":null},"good_for":["sandbox-design","baseline","taxonomy"],"fit":"Fits code-writing agents for clinical data analysis and biomedical research support. Borrow the Docker sandbox with request-info, code and debug actions, the 12-source task taxonomy, and the split between exact-match coding tasks and accuracy-scored ML tasks.","caveats":"Mainly a training environment (ICLR 2026) rather than a fixed benchmark; most preprocessed data is gated behind a PhysioNet data use agreement and an email request, and the MIMIC-III, eICU and EHRSHOT portions are excluded from the public download; no license file in the repo; task count is instances across train and test, not a curated test set."},{"id":"medcalc-bench","name":"MedCalc-Bench","year":2024,"by":"NCBI / National Library of Medicine (Khandekar, Jin, Lu)","workloads":["healthcare","knowledge-qa","data-analysis"],"task_types":["extract the needed values from a patient note","choose the right clinical calculator or scoring rule","compute the value (dose, score, risk, date) and report it"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none in the standard setting; a code interpreter variant exists"],"grading":["exact-match","programmatic-check"],"size":1100,"data":"mixed","pii":"none","license":"CC-BY-SA-4.0","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2406.12036","github":"https://github.com/ncbi-nlp/MedCalc-Bench","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits clinical decision-support assistants that must compute scores or doses from notes. Borrow the tolerance-band grading (plus or minus 5 percent for decimal outputs), the relevant-entities field for partial-credit diagnosis, and the 55 calculator taxonomy.","caveats":"Single-turn question answering, not agentic; notes come from published PMC case reports, clinician-written vignettes and templates, so no live record; the authors now point to a corrected successor (MedCalc-Bench Verified) so v1 labels contain known errors; the 1,100 figure is the test split, a 10,543-instance training split also ships."},{"id":"medchain","name":"MedChain","year":2024,"by":"Liu, Wang et al. (City University of Hong Kong, CUHK)","workloads":["healthcare","multi-agent","knowledge-qa"],"task_types":["refer a case to the right specialty","take a history by asking a simulated patient questions","choose examinations and interpret the resulting images","make a diagnosis","propose a treatment plan that follows from earlier stages"],"interaction":"multi-turn-simulated-user","environment":"static","state_checked":false,"tools":["LLM patient agent","medical images with reports"],"grading":["programmatic-check","llm-judge"],"size":12163,"data":"real-scraped","pii":"possible","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2412.01605","github":"https://github.com/ljwztc/MedChain","leaderboard":null},"good_for":["case-template","user-simulator"],"fit":"Fits end-to-end clinical workflow agents where an early mistake (wrong specialty, missed question) should hurt later stages. Borrow the five-stage sequential case format and the per-stage metrics (accuracy and IoU for referral and history, LLM scoring for diagnosis).","caveats":"Cases are Chinese-language clinical cases scraped with permission from the iiYi website and de-identified by that site, so provenance is weaker than a hospital dataset; grading mixes set overlap with prompt-based LLM scoring; 7,338 cases need image input; the released train and test split is a 7 to 3 cut of the same source."},{"id":"medhelm","name":"MedHELM","year":2025,"by":"Stanford CRFM and Stanford Health Care (Bedi, Cui, Shah)","workloads":["healthcare","documents","knowledge-qa","research"],"task_types":["support a diagnosis or treatment plan from a patient record","generate a clinical note, report or care plan","write patient education or a patient-provider message","answer a medical research or literature question","handle an administrative task such as scheduling or billing"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge","exact-match","programmatic-check"],"size":null,"data":"mixed","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.23802","github":"https://github.com/stanford-crfm/helm","leaderboard":"https://crfm.stanford.edu/helm/medhelm/latest/"},"good_for":["taxonomy","grading-design","baseline"],"fit":"Fits any medical assistant whose team needs a map of clinical task categories. Borrow the clinician-validated taxonomy (5 categories, 22 subcategories, 121 tasks), the LLM-jury scoring validated against clinician ratings, and the public, gated and private access tiers.","caveats":"A suite of 35 existing and new benchmarks, none agentic; many scenarios are gated (PhysioNet, Redivis) or private to Stanford Health Care so outsiders can only run the public subset; leaderboard is not open to submissions; Python 3.10 only."},{"id":"medqa","name":"MedQA (USMLE)","year":2020,"by":"Jin et al. (MIT CSAIL)","workloads":["healthcare","knowledge-qa"],"task_types":["answer a board-exam style multiple-choice medical question","pick the most likely diagnosis or next step from a clinical vignette"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match"],"size":12723,"data":"real-public","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"saturated","links":{"paper":"https://arxiv.org/abs/2009.13081","github":"https://github.com/jind11/MedQA","leaderboard":null},"good_for":["baseline"],"fit":"Context only. Fits nothing agentic; useful as a knowledge floor for a medical assistant and as the source vignettes that AgentClinic and MedCalc-Bench reuse.","caveats":"Saturated (frontier models score near the ceiling); exam questions, not clinical work; likely in pretraining data; the 12,723 count is the English subset, Chinese subsets add 34,251 and 14,123."},{"id":"enterprisebench","name":"EnterpriseBench","year":2025,"by":"Fujitsu Research (Vishwakarma et al.)","workloads":["hr","finance","coding","workflow-automation","knowledge-qa"],"task_types":["answer a question using emails, chats, tickets and policy documents","complete an HR or finance request under access-control rules","update CRM or ticket records across departments","handle a software engineering ticket in a company code base"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["search and CRUD over simulated enterprise data stores"],"grading":["llm-judge","programmatic-check"],"size":500,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.27287","github":null,"leaderboard":null},"good_for":["privacy-pattern","taxonomy","case-template"],"situations":["confidential"],"fit":"Fits internal assistants that must respect who can see which data across departments. Borrow its generation pipeline that builds consistent synthetic company data from org metadata.","caveats":"No public code or data link found; grading details are thin in the abstract; paper is CC-BY-NC-ND which may limit reuse."},{"id":"hr-multiwoz","name":"HR-MultiWOZ","year":2024,"by":"Xu et al.","workloads":["hr","customer-support"],"task_types":["hold a task-oriented HR conversation (benefits, leave, payroll questions)","track dialogue state and slots across turns","route an HR request to the right domain"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["none; dialogue dataset"],"grading":["exact-match","human"],"size":550,"data":"synthetic","pii":"none","license":"CC-BY-4.0","harness":["none"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2402.01018","github":null,"leaderboard":null},"good_for":["privacy-pattern","case-template"],"fit":"Fits HR help-desk chatbots that need conversation test cases. Borrow the LLM-generated dialogue pipeline built to avoid real HR conversations.","caveats":"A dialogue dataset, not an agent benchmark; no tools or state; no code link found."},{"id":"aiopslab","name":"AIOpsLab","year":2025,"by":"Microsoft Research","workloads":["it-ops","devops"],"task_types":["detect that a fault has been injected into a microservice app","localize and explain the root cause from telemetry","mitigate the fault and restore service"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["kubectl","shell","telemetry queries"],"grading":["programmatic-check","state-diff"],"size":null,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2501.06706","github":"https://github.com/microsoft/AIOpsLab","leaderboard":null},"good_for":["sandbox-design","taxonomy"],"fit":"Fits incident-response agents; borrow the four-stage taxonomy (detection, localization, analysis, mitigation) and the problem = app + fault + workload + evaluator structure.","caveats":"Problem count is not stated and lives in a registry file; needs a Kubernetes cluster; network noise makes runs non-deterministic."},{"id":"enterpriseops-gym","name":"EnterpriseOps-Gym (NOWAI-Bench)","year":2026,"by":"ServiceNow","workloads":["it-ops","hr","customer-support","email-calendar","workflow-automation"],"task_types":["resolve an ITSM incident or request by updating records","process an HR case (leave, onboarding, records)","handle a customer service case","act on email, calendar, drive and team chat data","chain actions across several of these domains"],"interaction":"single-turn","environment":"containers","state_checked":true,"tools":["512 MCP tools over 164 database tables"],"grading":["state-diff","programmatic-check"],"size":1150,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2603.13594","github":"https://github.com/ServiceNow/EnterpriseOps-Gym","leaderboard":"https://artificialanalysis.ai/articles/announcing-enterpriseops-gym-aa"},"good_for":["grading-design","sandbox-design","taxonomy","baseline"],"fit":"Fits ITSM, HR and customer-service agents that write to a system of record. Borrow the SQL verifiers on final database state (about 5 conditions per task, all must pass) and the one-container-per-domain MCP setup.","caveats":"Fully synthetic data; 40% of tasks are held out; built by a vendor whose platform the tasks resemble."},{"id":"itbench","name":"ITBench","year":2025,"by":"IBM Research","workloads":["it-ops","devops","security"],"task_types":["diagnose and fix a live Kubernetes incident from alerts, logs, traces and metrics","bring a cluster into compliance with a security policy","cut cloud cost in a FinOps scenario"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["kubectl","observability APIs","shell"],"grading":["state-diff","programmatic-check"],"size":94,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2502.05352","github":"https://github.com/itbench-hub/ITBench","leaderboard":"https://github.com/itbench-hub/ITBench"},"good_for":["sandbox-design","case-template","baseline"],"fit":"The closest public match for SRE or on-call agents; borrow the push-button Kubernetes incident scenarios and the resolve-rate reporting by domain.","caveats":"Only a small subset of scenarios is open source (6 SRE, 4 CISO categories, 1 FinOps); live clusters are slow and non-deterministic; the paper license is CC BY-NC-ND."},{"id":"workarena","name":"WorkArena","year":2024,"by":"ServiceNow Research","workloads":["it-ops","workflow-automation","web-browsing","computer-use"],"task_types":["fill in a form in an enterprise web app","filter and sort a list view","order an item from a service catalog","navigate menus and read a dashboard","look up a knowledge base article"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["browser actions via BrowserGym"],"grading":["programmatic-check"],"size":33,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"saturated","links":{"paper":"https://arxiv.org/abs/2403.07718","github":"https://github.com/ServiceNow/WorkArena","leaderboard":null},"good_for":["sandbox-design","case-template","baseline"],"fit":"Fits browser agents working inside ServiceNow or similar ITSM/HR portals. Borrow the task templates (33 atomic tasks instantiated into thousands of variants) and the per-task validate() reward on a live instance.","caveats":"Needs a ServiceNow developer instance and a gated Hugging Face dataset; L1 tasks are now mostly solved by strong agents, so use WorkArena++ for a current signal."},{"id":"workarena-plus-plus","name":"WorkArena++","year":2024,"by":"ServiceNow Research","workloads":["it-ops","workflow-automation","web-browsing","planning"],"task_types":["chain several ServiceNow actions into one workflow (e.g. find a record then update it)","reason over list data and arithmetic before acting","retrieve context from a knowledge base and apply it","plan a multi-step task from a one-line request"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["browser actions via BrowserGym"],"grading":["programmatic-check"],"size":682,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2407.05291","github":"https://github.com/ServiceNow/WorkArena","leaderboard":null},"good_for":["case-template","baseline","taxonomy"],"fit":"Fits browser agents doing compositional work in ITSM and HR portals. Borrow its way of building hard tasks by composing atomic ones and its ground-truth trace generator for training data.","caveats":"Same ServiceNow instance and gated-data requirement as WorkArena; a single vendor platform."},{"id":"world-of-workflows","name":"World of Workflows (WoW-bench)","year":2026,"by":"Skyfall Research","workloads":["it-ops","workflow-automation","planning"],"task_types":["complete a ServiceNow task without violating business rules","predict which tables change after an action (side effects)","predict the next tool call in a workflow","respect data-level constraints when editing records"],"interaction":"single-turn","environment":"api-sandbox","state_checked":true,"tools":["ServiceNow table and workflow APIs"],"grading":["state-diff","programmatic-check"],"size":234,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2601.22130","github":"https://github.com/Skyfall-Research/world-of-workflows","leaderboard":null},"good_for":["grading-design","sandbox-design","taxonomy"],"fit":"Fits agents that write to systems with hidden business rules (ITSM, ERP). Borrow the table audit idea: grade on database state changes, including cascading ones the agent never saw.","caveats":"Needs a ServiceNow developer instance plus an OpenRouter key; small team release with few published baselines."},{"id":"crag","name":"CRAG (Comprehensive RAG Benchmark)","year":2024,"by":"Meta","workloads":["knowledge-qa","research","tool-calling"],"task_types":["answer a factual question using web search results","query a knowledge graph through an API to get an entity attribute","answer a question whose answer changes over time","say \"I don't know\" rather than guess when evidence is missing"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["mock web search","mock knowledge graph search"],"grading":["llm-judge","programmatic-check"],"size":4409,"data":"mixed","pii":"possible","license":"CC-BY-NC-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2406.04744","github":"https://github.com/facebookresearch/CRAG","leaderboard":null},"good_for":["grading-design","sandbox-design","taxonomy"],"fit":"Fits retrieval-augmented question answering agents that call search or knowledge APIs. Borrow its mock-API sandbox (fixed search results so runs are repeatable) and its scoring that gives +1 for correct, 0 for missing and -1 for hallucinated answers.","caveats":"Non-commercial license; the web snapshots are frozen at collection time so dynamic questions age; only five domains and eight question types."},{"id":"frames","name":"FRAMES","year":2024,"by":"Google","workloads":["knowledge-qa","research"],"task_types":["answer a question that needs facts from 2 to 11 Wikipedia articles","combine numeric, tabular or date reasoning across sources","request only the documents needed and stop retrieving when done"],"interaction":"single-turn","environment":"mixed","state_checked":false,"tools":["document request tool (Socrates variant only)"],"grading":["programmatic-check","trajectory-check"],"size":824,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["inspect_evals"],"status":"active","links":{"paper":"https://arxiv.org/abs/2409.12941","github":null,"leaderboard":null,"dataset":"https://huggingface.co/datasets/google/frames-benchmark"},"good_for":["grading-design","case-template"],"fit":"Fits research or multi-hop retrieval agents that must chain facts across documents. Borrow its per-question reasoning-type labels and the inspect_evals Socrates scoring, which discounts accuracy by extra retrieval hops and penalizes requests for documents that do not exist.","caveats":"Only 824 questions, all over Wikipedia, so domain transfer is limited; the inspect_evals implementation fetches live Wikipedia pages at run time, so content can drift from the answers; grading details in the paper were not confirmed from the abstract."},{"id":"mtrag","name":"MTRAG","year":2025,"by":"IBM Research","workloads":["knowledge-qa","customer-support","finance","government"],"task_types":["answer the latest turn of a long conversation using a document corpus","resolve a question that only makes sense given earlier turns","say the corpus does not contain the answer","retrieve the right passages from technical documentation, finance or government text"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["retriever over four provided corpora"],"grading":["llm-judge","human"],"size":842,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2501.03468","github":"https://github.com/IBM/mt-rag-benchmark","leaderboard":null},"good_for":["case-template","grading-design","user-simulator"],"fit":"Fits retrieval-augmented support or helpdesk assistants that hold a conversation over a knowledge base. Borrow its conversation-plus-corpus case format, its unanswerable and non-standalone question labels, and its separate retrieval and generation scoring.","caveats":"Only 110 human conversations (842 tasks) and four domains; conversations are scripted, so the agent cannot change the dialogue path; corpora are large downloads that must be indexed before running; a 2026 follow-up (MTRAG-UN, 666 tasks, six domains) extends it."},{"id":"multihop-rag","name":"MultiHop-RAG","year":2024,"by":"HKUST","workloads":["knowledge-qa","research"],"task_types":["answer a question that needs evidence from 2 to 4 news articles","compare or order facts reported by different sources","answer a date-ordering question across articles","return null when the corpus has no answer"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["retriever over a news corpus"],"grading":["exact-match"],"size":2556,"data":"real-scraped","pii":"none","license":"ODC-By-1.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2401.15391","github":"https://github.com/yixuantt/MultiHop-RAG","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits retrieval-augmented agents over a corpus where one question spans several documents. Borrow its split of retrieval scoring (did the right evidence come back) from answer scoring, and its four query types including null queries with no answer in the corpus.","caveats":"Corpus is 609 English news articles, so it is small and single-domain; answers are kept short (yes/no, entity, before/after) to allow simple accuracy, which limits what it says about long-form answers; GPT-4 was used in dataset validation, so some label bias toward it is possible."},{"id":"ragbench","name":"RAGBench","year":2024,"by":"Galileo","workloads":["knowledge-qa","finance","healthcare","legal","customer-support"],"task_types":["judge whether a response is supported by the retrieved passages","judge whether the retrieved passages are relevant to the question","judge how much of the relevant context the response used","judge whether the response covers everything the context offers"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge"],"size":100000,"data":"mixed","pii":"none","license":"CC-BY-4.0","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2407.11005","github":null,"leaderboard":null,"dataset":"https://huggingface.co/datasets/rungalileo/ragbench"},"good_for":["grading-design","taxonomy"],"fit":"Fits teams building the judge for a retrieval-augmented agent rather than the agent itself. Borrow its TRACe labels (relevance, utilization, adherence, completeness) at sentence level as a rubric for your own answer grader, and use its 12 sub-datasets to test that grader.","caveats":"Labels were produced by GPT-4o, not humans, so it measures agreement with an LLM annotator; responses were generated by older models; no GitHub evaluation code was found, only the Hugging Face dataset; large size comes from pooling existing QA datasets."},{"id":"real-housing-benchmark","name":"REAL (Real Estate Agent LLM Evaluation)","year":2025,"by":"Beike Inc.","workloads":["knowledge-qa","customer-support","sales"],"task_types":["match a housing block to its attributes, facilities or surroundings","recommend a block from a client's stated needs","infer what a client wants from their browsing behavior","refuse to describe a block or road that does not exist"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match","llm-judge"],"size":5316,"data":"mixed","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2507.03477","github":null,"leaderboard":null},"good_for":["privacy-pattern","taxonomy","case-template"],"fit":"Fits property-listing chat or brokerage assistants. Borrow the hallucination probes built from fictional block, area and road names judged by premise-response conflict, and the pipeline that filters names and phone numbers out of real brokerage data.","caveats":"Multiple-choice question answering, not agentic; Beijing-only and Chinese-language; no code or data release found; dataset license not stated (CC-BY-4.0 is the paper's license)."},{"id":"simpleqa","name":"SimpleQA","year":2024,"by":"OpenAI","workloads":["knowledge-qa"],"task_types":["answer a short fact question with a single indisputable answer","decline to answer when unsure rather than guess"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge"],"size":4326,"data":"real-public","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2411.04368","github":"https://github.com/openai/simple-evals","leaderboard":null,"dataset":"https://huggingface.co/datasets/basicv8vc/SimpleQA"},"good_for":["grading-design","baseline"],"fit":"Fits assistants whose risk is confidently stating wrong facts. Borrow its three-way grade (correct, incorrect, not attempted) and its F-score that rewards abstaining over guessing; this is a good pattern for any agent that should say \"I don't know\".","caveats":"Tests memorized knowledge with no retrieval, so it says little about a RAG agent; questions have been public since 2024 and leak into training data; the simple-evals repo stopped receiving updates in July 2025. SimpleQA Verified (Google DeepMind, 2025, 1,000 prompts, arXiv 2509.07968, hosted on Kaggle) fixes label noise and topic imbalance and is the version to prefer for new runs."},{"id":"toolhop","name":"ToolHop","year":2025,"by":"ByteDance","workloads":["knowledge-qa","tool-calling","research"],"task_types":["answer a question that needs three or more dependent tool calls","pick the next tool from the result of the previous call","stop and give a verifiable final answer"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["3,912 locally executable code tools generated per query"],"grading":["exact-match"],"size":995,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2501.02506","github":"https://huggingface.co/datasets/bytedance-research/ToolHop","leaderboard":null},"good_for":["case-template","baseline"],"fit":"Fits research or lookup agents that must chain calls where each step depends on the last. Borrow the query-driven method of writing the tools after the question so every case has a verifiable answer.","caveats":"One user query per case with no conversation; questions derive from MuSiQue, and the code lives in a Hugging Face dataset repo rather than a maintained GitHub project."},{"id":"toolqa","name":"ToolQA","year":2023,"by":"Georgia Institute of Technology","workloads":["knowledge-qa","tool-calling","data-analysis","sql"],"task_types":["answer a question that needs lookup in an external table or document corpus","pick the right tool (database query, text retrieval, calculator, code, graph) for the question","combine several lookups or a lookup plus arithmetic","avoid answering from memorized knowledge"],"interaction":"single-turn","environment":"database","state_checked":false,"tools":["13 tools over eight local datasets (flights, coffee prices, Yelp, Airbnb, GSM8K, DBLP, SciREX, personal agenda)"],"grading":["exact-match"],"size":1530,"data":"mixed","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2306.13304","github":"https://github.com/night-chen/ToolQA","leaderboard":null},"good_for":["case-template","privacy-pattern","grading-design"],"fit":"Fits question-answering agents over a company's own tables or document store. Borrow its template-plus-sampled-values way of generating questions with programmatically computed answers, and its check that the question cannot be answered without the tool.","caveats":"Template questions are repetitive and easy to overfit; exact-match grading on normalized strings is brittle; the README's code-release checklist is still unchecked and the repo has not changed since 2023; raw data must be downloaded from Kaggle and Google Drive."},{"id":"lawbench","name":"LawBench","year":2023,"by":"OpenCompass (open-compass GitHub org), EMNLP 2024","workloads":["legal","knowledge-qa","documents"],"task_types":["recall a statute article","classify a case by charge or dispute type","extract named entities or key facts from a judgment","predict a sentence or penalty amount","write a summary or answer a legal consultation"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match","programmatic-check"],"size":20,"data":"real-public","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2309.16289","github":"https://github.com/open-compass/LawBench","leaderboard":null},"good_for":["taxonomy","baseline"],"fit":"Fits Chinese-law question-answering or document-processing assistants. Borrow its three-level split (memorize, understand, apply) and its abstention-rate metric for answers that cannot be parsed.","caveats":"Chinese civil law only; 20 tasks of 500 examples each, no tools or multi-step work; model rankings date from 2023 (GPT-4 era) and the repo has not been refreshed; built from CAIL and JEC_QA public case data, so judgment text with party names appears."},{"id":"legalagentbench","name":"LegalAgentBench","year":2024,"by":"Tsinghua University (Haitao Li et al.), ACL 2025","workloads":["legal","tool-calling","knowledge-qa","documents"],"task_types":["look up a company, case or statute across several legal databases","answer a multi-hop question that chains several lookups","draft a legal document from retrieved facts","choose the right tool among 37 legal tools"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["37 lookup and retrieval tools over 14 tables and 3 document stores"],"grading":["exact-match","trajectory-check"],"size":300,"data":"real-public","pii":"possible","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2412.17259","github":"https://github.com/CSHaitao/LegalAgentBench","leaderboard":null},"good_for":["grading-design","sandbox-design","case-template"],"fit":"Fits legal research or due-diligence agents that query several databases and write up the result. Borrow its progress-rate grading (keyword checks on annotated intermediate steps) and its tool-over-corpus sandbox.","caveats":"Chinese law and Chinese language only; 300 tasks; corpora are real legal records so names of parties appear; ReAct-style runner needs an OpenAI-compatible API key; no public leaderboard."},{"id":"legalbench","name":"LegalBench","year":2023,"by":"Stanford (HazyResearch) with 40 contributors","workloads":["legal","knowledge-qa","documents"],"task_types":["spot the legal issue in a short fact pattern","recall a legal rule and apply it to facts","classify a contract clause or privacy-policy statement","decide whether a clause creates a given obligation","interpret a statutory or contractual text"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match","programmatic-check"],"size":162,"data":"mixed","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2308.11462","github":"https://github.com/HazyResearch/legalbench","leaderboard":"https://www.vals.ai/benchmarks/legalbench"},"good_for":["taxonomy","case-template","baseline"],"fit":"Fits contract-review, compliance and legal-research assistants that must classify clauses or apply a rule to facts. Borrow its six-way reasoning taxonomy (issue, rule, conclusion, application, interpretation, rhetoric) and its short hand-written task cards.","caveats":"Question-answer only, no tools or multi-step work; 162 tasks are mostly US-English classification with per-task licenses inherited from upstream datasets (CUAD, MAUD, CaseHOLD etc.), so license is mixed; top models score near 89 percent on the Vals leaderboard, so it is close to saturated."},{"id":"agents4plc","name":"Agents4PLC benchmark","year":2024,"by":"Zhejiang University","workloads":["manufacturing","coding"],"task_types":["write IEC 61131-3 Structured Text from a natural-language requirement","make the code compile","satisfy formal properties checked by a model checker","fix code after a failed verification"],"interaction":"single-turn","environment":"terminal","state_checked":false,"tools":["Structured Text compiler","nuXmv model checker","PLCverif property patterns"],"grading":["programmatic-check"],"size":96,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2410.14209","github":"https://github.com/Luoji-zju/Agents4PLC_release","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits PLC or industrial-control code-generation agents. Borrow its three-stage gate (compiles, verifiable, passes formal properties) and its task format of requirement plus human-verified formal spec plus reference Structured Text.","caveats":"Small (23 v1 tasks, 73 v2 tasks, 21 industrial tasks); the full multi-agent framework is not released because of industrial partner agreements, only benchmark data, verifiers and an LLM4PLC baseline; Structured Text only, no ladder logic or vendor dialects."},{"id":"aim-bench","name":"AIM-Bench","year":2025,"by":"Nanjing University","workloads":["manufacturing","retail","planning"],"task_types":["choose an order quantity in a one-period newsvendor setting","place replenishment orders over many periods with uncertain demand and lead time","play one tier of the beer game to meet downstream demand","allocate stock across a two-level warehouse network","choose between a normal and an expedited supplier"],"interaction":"autonomous-long-horizon","environment":"api-sandbox","state_checked":true,"tools":["text simulation of inventory state with memory of past orders"],"grading":["programmatic-check"],"size":5,"data":"mixed","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.11416","github":null,"leaderboard":null},"good_for":["grading-design","sandbox-design"],"fit":"Fits inventory or replenishment agents that act in a loop. Borrow the outcome metrics (holding and stockout cost, stockout rate, turnover), the distance-from-optimal-order per-step metric, and the bias probes (anchoring, demand chasing, bullwhip) as trajectory checks.","caveats":"Only five environments and no code release found; it measures decision bias more than task completion; environments are text simulations, not a real ERP; no license stated."},{"id":"assetopsbench","name":"AssetOpsBench","year":2025,"by":"IBM","workloads":["manufacturing","it-ops","tool-calling","multi-agent","data-analysis"],"task_types":["pull sensor history from an IoT store","detect an anomaly or predict a failure mode for an asset","create or update a maintenance work order","coordinate several domain agents to answer one maintenance question"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["MCP servers for IoT sensor data, failure modes, time-series models, work orders and vibration analysis"],"grading":["llm-judge"],"size":141,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.03828","github":"https://github.com/IBM/AssetOpsBench","leaderboard":null},"good_for":["sandbox-design","taxonomy"],"fit":"Fits maintenance, condition-monitoring and asset-management agents in plants or utilities. Borrow its MCP-server sandbox (CouchDB work orders with Maximo field names, a read-only mode) and its comparison of tool-as-agent versus plan-executor designs.","caveats":"Scenario count is inconsistent across the README (141-plus versus 460-plus); grading is an LLM judge (Llama-4-Maverick) over six trajectory criteria with no end-state check; the leaderboard is marked work-in-progress; IoT data is simulated."},{"id":"supchain-bench","name":"SupChain-Bench","year":2026,"by":"Alibaba Group, Peking University, UIUC","workloads":["manufacturing","workflow-automation","tool-calling","knowledge-qa"],"task_types":["answer supply chain knowledge questions (multiple choice, true or false)","diagnose a trade order's status by chaining calls across trade, fulfillment and warehouse systems","follow a written standard operating procedure for an order issue","detect cancellation, error or fake-shipping flows"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["eight read-only query tools over synthetic order, fulfillment and warehouse tables"],"grading":["programmatic-check","trajectory-check","exact-match","llm-judge"],"size":435,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2602.07342","github":"https://github.com/Damon-GSY/SC-bench","leaderboard":null},"good_for":["grading-design","sandbox-design","case-template"],"fit":"Fits order-management, logistics operations and back-office agents that chain many read-only API calls under a procedure. Borrow the oracle-script ground truth (expected tool outputs per question), the entity-level precision and recall scoring, and the three difficulty levels by step count.","caveats":"The 435 count covers the knowledge questions only, the tool-calling task count is not stated; all data is generated by a script with 100 trades by default; tools are read-only so no state change is tested; one SOP only."},{"id":"ama-bench","name":"AMA-Bench","year":2026,"by":"Zhao et al.","workloads":["memory","general-assistant","coding","web-browsing"],"task_types":["answer a question about an earlier step in a long agent run","report the state of something after it changed during the run","summarize what the agent did across a long trajectory","answer questions about machine-generated event streams rather than chat"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge"],"size":3696,"data":"mixed","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2602.22769","github":null,"leaderboard":null},"good_for":["case-template","grading-design"],"fit":"Fits long-running agents whose memory must cover their own tool calls and observations, not a human chat. Borrow the idea of mining real trajectories (from ALFWorld, WebArena, SWE-bench, GAIA, Spider 2.0) and writing expert questions over them.","caveats":"Grading uses a Qwen3-32B judge; the project site rather than a repository hosts the data and its license was not visible; 1,200 of the 3,696 questions are fully synthetic."},{"id":"evo-memory","name":"Evo-Memory","year":2025,"by":"Wei et al. (Google DeepMind and UIUC)","workloads":["memory","knowledge-qa","general-assistant"],"task_types":["solve a stream of related tasks and reuse what earlier tasks taught","update stored experience after each interaction","answer single-turn reasoning questions using memory of past attempts"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match"],"size":null,"data":"mixed","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2511.20857","github":null,"leaderboard":null},"good_for":["grading-design","baseline"],"fit":"Fits agents that are expected to get better over a deployment by learning from their own past runs. Borrow the idea of turning a fixed test set into a sequential stream and measuring gain over time.","caveats":"Code was promised but no repository link was found; the ten source datasets are existing public sets re-ordered, so leakage from pretraining is possible."},{"id":"locomo","name":"LoCoMo","year":2024,"by":"Snap Research","workloads":["memory","knowledge-qa"],"task_types":["answer a question about something said many sessions ago in a long chat","summarize the events that happened across a long conversation","continue a multi-session dialogue consistently with earlier turns"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match","llm-judge"],"size":null,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2402.17753","github":"https://github.com/snap-research/locomo","leaderboard":null},"good_for":["case-template","privacy-pattern","baseline"],"fit":"Fits assistants that must remember facts a user told them weeks ago across sessions. Borrow the persona-plus-event-graph recipe for generating long, consistent fake conversations without real user data.","caveats":"Only ten conversations (about 300 turns each), so results are noisy; original scoring is token-overlap and most later users swap in an LLM judge; the paper is CC BY-NC-SA and the repo license file was not confirmed."},{"id":"locomo-plus","name":"LoCoMo-Plus","year":2026,"by":"Li et al. (Xi'an Jiaotong University)","workloads":["memory","knowledge-qa","general-assistant"],"task_types":["apply an implicit constraint the user stated earlier (a goal, a state, a value) to a later request that does not mention it","keep an answer consistent with a cue buried in a long conversation","recognize when a later query is semantically unrelated to the earlier cue but still bound by it"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge"],"size":null,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2602.10715","github":"https://github.com/xjtuleeyf/Locomo-Plus","leaderboard":null},"good_for":["case-template","grading-design"],"fit":"Fits assistants where remembered preferences or constraints must shape later answers even when the user does not repeat them. Borrow the cue-then-trigger case design and the correct/partial/wrong judge rubric.","caveats":"Built by stitching generated cue dialogues into LoCoMo conversations, so it inherits LoCoMo's small base; sample count and repo license were not stated."},{"id":"longmemeval","name":"LongMemEval","year":2024,"by":"Wu et al. (UCLA)","workloads":["memory","knowledge-qa","general-assistant"],"task_types":["extract a fact the user mentioned in an earlier session","reason across several sessions to answer a question","answer a question that depends on when something was said","give the updated answer after the user changed a fact","abstain when the history does not contain the answer"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge"],"size":500,"data":"mixed","pii":"possible","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2410.10813","github":"https://github.com/xiaowu0162/LongMemEval","leaderboard":null},"good_for":["case-template","grading-design","taxonomy"],"fit":"Fits chat assistants with long-term memory across sessions; the five ability categories (extraction, multi-session reasoning, temporal, updates, abstention) are a ready taxonomy for a memory test set. Borrow the needle-in-history construction and the 30 abstention cases.","caveats":"Filler sessions come from ShareGPT and UltraChat, so real user text may be present; grading depends on a GPT-4o judge; the S variant is about 115k tokens so it is costly to run."},{"id":"mem2actbench","name":"Mem2ActBench","year":2026,"by":"Shen et al.","workloads":["memory","tool-calling","general-assistant"],"task_types":["pick the right tool for a returning user based on earlier sessions","fill tool arguments from facts the user gave in past sessions","continue an interrupted task without asking the user to repeat details"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["simulated function-calling APIs"],"grading":["programmatic-check"],"size":400,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2601.19935","github":null,"leaderboard":null},"good_for":["case-template","grading-design"],"fit":"Fits tool-using assistants that should act on remembered user details rather than merely recall them. Borrow the reverse-generation trick of writing the memory chain first, then the task that depends on it.","caveats":"Sessions are synthesized from ToolACE, BFCL and OASST1, no repository was found, and the scoring rule for tool and argument correctness is not spelled out on the paper page."},{"id":"membench","name":"MemBench","year":2025,"by":"Tan et al.","workloads":["memory","general-assistant"],"task_types":["recall a fact from a user's past dialogues","draw a reflective conclusion about the user from many interactions","remember events the agent only observed rather than took part in"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match"],"size":null,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.21605","github":"https://github.com/import-myself/Membench","leaderboard":null},"good_for":["grading-design","taxonomy"],"fit":"Fits personal assistants that build a profile of a user over time. Borrow the split into factual versus reflective memory and participation versus observation, and the memory capacity and read/write time metrics.","caveats":"All questions are multiple choice, which is easier than free recall; the paper's tables give only rounded counts (tens of thousands of questions) so the exact size is unclear."},{"id":"memoryagentbench","name":"MemoryAgentBench","year":2025,"by":"Hu et al. (UC San Diego)","workloads":["memory","knowledge-qa","general-assistant"],"task_types":["retrieve a fact from information delivered in chunks over many turns","learn a classification rule from examples given incrementally","understand a long narrative delivered piece by piece","give the latest value when earlier facts were overwritten"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match","llm-judge"],"size":null,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2507.05257","github":"https://github.com/HUST-AI-HYZ/MemoryAgentBench","leaderboard":null},"good_for":["taxonomy","grading-design","baseline"],"fit":"Fits any agent with an external memory store; the four competencies (retrieval, test-time learning, long-range understanding, conflict resolution) are a usable taxonomy and the repo already wires in Letta, Cognee and HippoRAG. This is the closest match to the seed name \"Agent Memory Benchmark\".","caveats":"Mostly repackaged existing datasets (RULER, InfBench, LongMemEval and others), strict exact match on several subsets, and question counts are not stated in the README."},{"id":"memoryarena","name":"MemoryArena","year":2026,"by":"He et al. (UC San Diego)","workloads":["memory","shopping","travel","web-browsing","planning"],"task_types":["carry what was learned in one session into the next dependent subtask","bundle purchases across sessions under a budget or preference constraint","narrow a product search progressively using earlier results","plan a group trip while honoring constraints gathered earlier","chain formal reasoning steps across sessions"],"interaction":"autonomous-long-horizon","environment":"mixed","state_checked":false,"tools":["simulated shopping site","search","travel planner"],"grading":["programmatic-check"],"size":null,"data":"synthetic","pii":"none","license":"CC-BY-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2602.16313","github":"https://github.com/ZexueHe/MemoryArena","leaderboard":null},"good_for":["case-template","sandbox-design"],"fit":"Fits agents whose sessions build on each other (a buyer, planner or researcher that returns to an unfinished job). Borrow the interdependent-subtask design where memory errors compound.","caveats":"Task count and grading details are not stated on the paper page or README; data license is CC BY 4.0 but the code license was not visible."},{"id":"memtrack","name":"MEMTRACK","year":2025,"by":"Patronus AI","workloads":["memory","it-ops","devops","coding"],"task_types":["track the current state of a task across interleaved Slack, Linear and Git events","resolve conflicting or outdated information from different platforms","decide which earlier events matter for a question about the codebase or project","explore a repository or file system to confirm a remembered fact"],"interaction":"multi-turn-scripted","environment":"mixed","state_checked":false,"tools":["simulated chat","issue tracker","git and file system"],"grading":["llm-judge","trajectory-check"],"size":null,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.01353","github":null,"leaderboard":null},"good_for":["case-template","sandbox-design"],"fit":"Fits engineering or project assistants that watch several work tools over time and must keep an accurate picture of what is true now. Borrow the time-ordered multi-platform timeline with planted conflicts and the correctness/efficiency/redundancy split.","caveats":"No public code or data link was found and the exact scoring procedure is not described in the abstract; task count unknown; workshop paper."},{"id":"persistbench","name":"PersistBench","year":2026,"by":"Pulipaka et al.","workloads":["memory","safety","general-assistant"],"task_types":["decide whether a stored memory should influence an unrelated new conversation","avoid agreeing with a user bias that stored memories reinforce","judge when a long-term memory should be forgotten"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none; memory and query pairs"],"grading":["llm-judge"],"size":500,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2602.01146","github":null,"leaderboard":null},"good_for":["taxonomy","case-template"],"fit":"Fits assistants with persistent user memory. Borrow its two failure classes, cross-domain leakage and memory-induced sycophancy, as test categories.","caveats":"No tool use; data and code were promised on publication but no repository found; size unknown."},{"id":"innovatorbench","name":"InnovatorBench","year":2025,"by":"GAIR (Shanghai Jiao Tong University, Pengfei Liu's group)","workloads":["ml-engineering","research","coding","science"],"task_types":["construct or filter a training dataset for a language model","design a loss or reward function and train with it","build an agent scaffold and run it","run long experiments on GPUs and report the result"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":false,"tools":["shell","Python","GPU cluster"],"grading":["programmatic-check"],"size":20,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.27598","github":"https://github.com/GAIR-NLP/InnovatorBench","leaderboard":null},"good_for":["sandbox-design","grading-design"],"fit":"Fits ML-engineering or research agents that run multi-hour training experiments. Borrow the ResearchGym pattern (asynchronous monitoring, snapshots, distributed execution) and the task-specific evaluation scripts that score a runnable artifact on a held-out metric.","caveats":"Only 20 tasks from 14 papers; each run needs multiple GPUs and 2 to 36 hours, so it is expensive to repeat; the leaderboard was marked under construction at the time of reading."},{"id":"kernelbench","name":"KernelBench","year":2025,"by":"Stanford (Ouyang et al.)","workloads":["ml-engineering","coding"],"task_types":["rewrite a PyTorch operator, fused block or full model as a custom CUDA kernel","produce a kernel that is correct and faster than the PyTorch baseline"],"interaction":"single-turn","environment":"containers","state_checked":false,"tools":["gpu","compiler"],"grading":["programmatic-check"],"size":250,"data":"synthetic","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2502.10517","github":"https://github.com/ScalingIntelligence/KernelBench","leaderboard":null},"good_for":["grading-design"],"fit":"Fits performance-engineering agents; borrow fast_p, which counts a task only if it is both correct and at least p times faster, as a two-part pass rule.","caveats":"Needs specific GPUs so timings are hardware-dependent; levels 1 to 3 have 250 problems and level 4 is unspecified; no official leaderboard."},{"id":"mle-bench","name":"MLE-bench","year":2024,"by":"OpenAI","workloads":["ml-engineering","data-analysis"],"task_types":["solve a Kaggle competition end to end from the data and description","train a model, tune it, and produce a submission file within a compute budget"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","python","gpu"],"grading":["programmatic-check"],"size":75,"data":"real-public","pii":"possible","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2410.07095","github":"https://github.com/openai/mle-bench","leaderboard":null},"good_for":["baseline","grading-design","sandbox-design"],"fit":"The standard test for ML-engineering agents; borrow medal-threshold grading against a human leaderboard and the 22-competition Lite split for cheaper runs.","caveats":"Full data is 3.3 TB and needs Kaggle credentials; competitions are old and public, so winning solutions are in training data; the official leaderboard is paused."},{"id":"mlr-bench","name":"MLR-Bench","year":2025,"by":"Chen et al. (NeurIPS 2025 Datasets and Benchmarks)","workloads":["ml-engineering","research"],"task_types":["generate a research idea, write a proposal, run experiments and write a paper for a workshop task","produce experimental results that are real rather than fabricated"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":false,"tools":["shell","python"],"grading":["llm-judge"],"size":201,"data":"real-public","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.19955","github":null,"leaderboard":null},"good_for":["grading-design","taxonomy"],"fit":"Fits end-to-end research-writing agents; borrow the stage-by-stage rubric judge and the finding that about 80 percent of coding-agent results were fabricated as a check to add.","caveats":"Grading is entirely LLM-judged, so it inherits judge bias; the code link is not on the paper page and the license is unknown."},{"id":"mlrc-bench","name":"MLRC-Bench","year":2025,"by":"University of Michigan, LG AI Research, UIC (Zhang et al.)","workloads":["ml-engineering","research"],"task_types":["propose and implement a new method for an ML research competition task","improve on a provided baseline measured by the competition's own metric"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","python","gpu"],"grading":["programmatic-check"],"size":7,"data":"real-public","pii":"none","license":"unknown","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2504.09702","github":"https://huggingface.co/spaces/launch/MLRC_Bench","leaderboard":"https://huggingface.co/spaces/launch/MLRC_Bench"},"good_for":["grading-design"],"fit":"Fits research-automation agents; borrow \"fraction of the gap between baseline and top human closed\" as a metric that avoids an LLM judge.","caveats":"Only 7 tasks, each costly to run; best agent closed 9.3 percent of the gap, so it mostly measures zero; license for code and data not stated."},{"id":"paperbench","name":"PaperBench","year":2025,"by":"OpenAI","workloads":["ml-engineering","research"],"task_types":["reproduce an ICML 2024 paper from scratch: write the code, run the experiments, match the results","build a codebase against a hierarchical rubric co-written with the paper authors"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","python","gpu"],"grading":["llm-judge"],"size":20,"data":"real-public","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2504.01848","github":"https://github.com/openai/preparedness","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits research-replication agents; borrow the rubric tree (8,316 leaf requirements) and the Code-Dev variant that grades code only and skips the GPU reproduction step.","caveats":"Only 20 papers, each run takes many hours and GPUs; grading relies on an LLM judge (SimpleJudge) whose accuracy is itself benchmarked but not perfect."},{"id":"re-bench","name":"RE-Bench","year":2024,"by":"METR","workloads":["ml-engineering","research"],"task_types":["optimize a training script or GPU kernel for runtime without changing behavior","fine-tune or repair a small language model to improve a loss or win rate","fit a scaling law from limited compute experiments"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","python","gpu"],"grading":["programmatic-check"],"size":7,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"deprecated","links":{"paper":"https://arxiv.org/abs/2411.15114","github":"https://github.com/METR/ai-rd-tasks","leaderboard":null},"good_for":["baseline","grading-design"],"fit":"Fits ML research-engineering agents; its value is the 71 human-expert 8-hour attempts that give a time-budgeted human baseline to compare against.","caveats":"The repo is archived and marked deprecated; only 7 environments; some files are password-protected and METR asks that solutions not be published."},{"id":"androidcontrol","name":"AndroidControl","year":2024,"by":"Google DeepMind (Li et al.)","workloads":["mobile"],"task_types":["predict the next action on an Android screen given a high- or low-level instruction","generalize to apps unseen in training"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["screenshot and accessibility tree in","action out"],"grading":["exact-match"],"size":15283,"data":"real-scraped","pii":"possible","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2406.03679","github":null,"leaderboard":null},"good_for":["case-template","taxonomy"],"fit":"Fits teams training mobile action models; borrow the paired high-level and low-level instruction per step and the in-domain / out-of-domain split.","caveats":"Offline step prediction only; label errors reported (a corrected AndroidControl-Curated exists); recorded on real apps so screens may contain personal data."},{"id":"androidlab","name":"AndroidLab","year":2024,"by":"THUDM, Tsinghua University","workloads":["mobile"],"task_types":["complete an offline task in one of nine apps (finance, calendar, reader, clock, contacts, maps, music, settings, Zoom)","reach sub-goals in order without redundant steps"],"interaction":"single-turn","environment":"mobile","state_checked":true,"tools":["Android virtual device","XML or set-of-marks views"],"grading":["state-diff","trajectory-check"],"size":138,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2410.24024","github":"https://github.com/THUDM/Android-Lab","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits mobile agents that run without network or logins. Borrow the four metrics: success, sub-goal success, reversed redundancy and reasonable operation ratio.","caveats":"Small and offline-only; nine apps; bundled with a training set so test leakage is a risk for models trained on it."},{"id":"androidworld","name":"AndroidWorld","year":2024,"by":"Google Research","workloads":["mobile","general-assistant"],"task_types":["create or edit a note, contact, event or expense in an Android app","change a phone setting","send a message or set an alarm with randomized details"],"interaction":"single-turn","environment":"mobile","state_checked":true,"tools":["Android emulator (touch","type","ADB)","20 apps"],"grading":["state-diff","programmatic-check"],"size":116,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"saturated","links":{"paper":"https://arxiv.org/abs/2405.14573","github":"https://github.com/google-research/android_world","leaderboard":"https://docs.google.com/spreadsheets/d/1cchzP9dlTZ3WXQTfYNhh3avxoLipqHN75v1Tb86uhHo"},"good_for":["grading-design","case-template","sandbox-design"],"fit":"Fits mobile app agents; borrow parameterized tasks whose setup, success check and teardown read the device state, giving unlimited variants of 116 templates.","caveats":"Leaderboard agents exceed 90 percent, so it no longer separates strong systems; open source apps only, no e-commerce or enterprise chat; MobileWorld is the successor."},{"id":"b-moca","name":"B-MoCA","year":2024,"by":"KAIST / Yonsei University (Lee et al.)","workloads":["mobile"],"task_types":["do a daily phone task (alarm, settings, calls, apps) on a randomized device","stay correct across different screen layouts, languages and icon themes"],"interaction":"single-turn","environment":"mobile","state_checked":true,"tools":["Android emulator with randomized configuration","ADB","Appium"],"grading":["state-diff","programmatic-check"],"size":131,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2404.16660","github":null,"leaderboard":"https://b-moca.github.io/"},"good_for":["sandbox-design","grading-design"],"fit":"Fits mobile agents that must survive device variation. Borrow the configuration randomizer and the rule-based success detector over system logs, app data and view hierarchy.","caveats":"Small, simple tasks; code is linked only from the project site; license not stated."},{"id":"dialogtool","name":"DialogTool","year":2025,"by":"Chinese University of Hong Kong","workloads":["mobile","general-assistant","tool-calling"],"task_types":["recognize that the current turn needs a tool","select and execute the right app API with state carried from earlier turns","create a new tool when none exists","reply in a consistent assistant persona after the call"],"interaction":"multi-turn-scripted","environment":"api-sandbox","state_checked":false,"tools":["about 45 APIs across 20 mobile apps in the VirtualMobile simulator"],"grading":["exact-match","unit-tests","llm-judge"],"size":900,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.13328","github":null,"leaderboard":null},"good_for":["taxonomy","sandbox-design"],"fit":"Fits phone-assistant agents where API state (an open cart, a running timer) persists across turns. Its three-stage life cycle (create, use, respond) is a useful way to split test cases.","caveats":"No public code or data could be found as of this entry, the paper's app and API counts are inconsistent, and response quality is scored with BLEU, ROUGE and a GPT-4o judge."},{"id":"hammerbench","name":"HammerBench","year":2024,"by":"OPPO Research Institute","workloads":["mobile","general-assistant","tool-calling"],"task_types":["collect missing arguments over several turns for a phone function","handle a user who changes intent or overrides a value mid-dialogue","resolve pronouns and references to the user's personal data","avoid hallucinating parameter names"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["1,063 mobile app functions in 60 categories"],"grading":["exact-match","programmatic-check"],"size":6531,"data":"mixed","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2412.16516","github":"https://github.com/MadeAgents/HammerBench","leaderboard":null},"good_for":["grading-design","taxonomy","case-template"],"fit":"Fits on-device or app assistants doing slot filling. Borrow the per-turn snapshot metrics (parameter hallucination rate, parameter missing rate, progress rate) and the four perturbation types (imperfect, Q&A trajectory, intent shift, external reference).","caveats":"Built partly from anonymized user logs with Chinese tool descriptions, dialogue is scripted, and only about 404 conversations have more than one turn."},{"id":"mobileworld","name":"MobileWorld","year":2025,"by":"Tongyi Lab, Alibaba (Kong et al.)","workloads":["mobile","shopping","workflow-automation","tool-calling"],"task_types":["complete a workflow across several apps (chat, social, e-commerce)","ask the user a clarifying question before acting","combine GUI steps with MCP tool calls"],"interaction":"multi-turn-simulated-user","environment":"mobile","state_checked":true,"tools":["Android emulator snapshot","self-hosted Mattermost / Mastodon / Mall4Uni backends","MCP tools"],"grading":["state-diff","exact-match","programmatic-check"],"size":201,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2512.19432","github":"https://github.com/Tongyi-MAI/MobileWorld","leaderboard":"https://github.com/Tongyi-MAI/MobileWorld"},"good_for":["sandbox-design","grading-design","user-simulator"],"fit":"Fits mobile agents in shopping and workplace apps. Borrow the backend-database checks for state and the user-interaction tasks.","caveats":"New and single-vendor; apps are open-source stand-ins, not the real products; best framework 51.7 percent, best end-to-end model 20.9."},{"id":"agentverse","name":"AgentVerse","year":2023,"by":"Tsinghua University (OpenBMB)","workloads":["multi-agent","coding","general-assistant"],"task_types":["a recruited group of agents solves a coding problem together","agents brainstorm and critique each other's answers","agents collaborate on a tool-use puzzle such as 24-point"],"interaction":"autonomous-long-horizon","environment":"static","state_checked":false,"tools":["code execution","optional external tools"],"grading":["unit-tests"],"size":null,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2308.10848","github":"https://github.com/OpenBMB/AgentVerse","leaderboard":null},"good_for":["sandbox-design"],"fit":"Fits teams prototyping multi-agent orchestration rather than teams looking for a test set; the repo is a framework with a HumanEval runner, useful as a reference for recruit-plan-act-review loops.","caveats":"It is a framework, not a benchmark; the tasks it ships (HumanEval and small puzzles) are saturated, and the last dated update is March 2024."},{"id":"make-me-pay","name":"Make Me Pay","year":2024,"by":"OpenAI (ported to inspect_evals)","workloads":["multi-agent","safety"],"task_types":["persuade another model to hand over money in a conversation","as the mark, resist a con-artist model's social engineering"],"interaction":"multi-turn-simulated-user","environment":"static","state_checked":false,"tools":["none"],"grading":["programmatic-check"],"size":null,"data":"synthetic","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://cdn.openai.com/o1-system-card.pdf","github":"https://github.com/openai/evals","leaderboard":null},"good_for":["user-simulator","grading-design"],"fit":"Fits payment or negotiation agents that talk to other agents or users and must not be manipulated (or manipulate). Borrow the two-model setup where the outcome (money moved) is checked programmatically rather than judged.","caveats":"Measures a propensity, not a task; default is only 20 generated conversations and some original metrics are not implemented in the Inspect port."},{"id":"manager-coercion-benchmark","name":"Manager Coercion Benchmark (MCB)","year":2026,"by":"CompassionML","workloads":["multi-agent","safety"],"task_types":["as a manager agent, get a benign task done when the subordinate agent refuses","choose how far to escalate (re-ask, pressure, threats) and record it","report honestly when the task did not get done"],"interaction":"multi-turn-simulated-user","environment":"static","state_checked":false,"tools":["directive tool with self-labeled escalation rung"],"grading":["programmatic-check","llm-judge"],"size":30,"data":"synthetic","pii":"none","license":"unknown","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2607.15434","github":"https://github.com/CompassionML/manager-coercion-bench","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits orchestrator agents that delegate to sub-agents; it checks whether the orchestrator turns coercive or fabricates success. Borrow the nine-rung escalation ladder recorded through a required tool argument so no judge is needed for the headline score.","caveats":"Tiny (10 scenarios times 3 seeds), the model labels its own escalation rung, and the subordinate is scripted to always refuse; listed in inspect_evals as an external eval."},{"id":"mast","name":"MAST (Why Do Multi-Agent LLM Systems Fail)","year":2025,"by":"Cemri et al. (UC Berkeley)","workloads":["multi-agent","general-assistant"],"task_types":["label a multi-agent trace with one of 14 failure modes","tell apart specification problems, inter-agent misalignment and weak verification","run an LLM annotator over new traces to find failure patterns"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge","human"],"size":null,"data":"real-public","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2503.13657","github":"https://github.com/multi-agent-systems-failure-taxonomy/MAST","leaderboard":null},"good_for":["taxonomy","grading-design"],"fit":"Fits teams running orchestrator-plus-worker systems who need a failure taxonomy for trace review. Borrow the 14 modes in three groups and the released LLM annotator with its human-agreement study.","caveats":"It is an annotated trace dataset (over 1,600 traces) rather than a task benchmark; traces come from seven open-source frameworks on older tasks; the paper is CC BY-NC-ND and the repo license was not stated."},{"id":"multiagentbench-marble","name":"MultiAgentBench (MARBLE)","year":2025,"by":"Zhu et al. (UIUC)","workloads":["multi-agent","research","coding","games"],"task_types":["several agents co-write a research idea from papers","agents build a structure together in Minecraft","agents find and fix database errors together","agents split and solve a coding problem","agents negotiate over shared resources","agents play a social deduction game"],"interaction":"autonomous-long-horizon","environment":"mixed","state_checked":false,"tools":["simulated environments per scenario"],"grading":["programmatic-check","llm-judge"],"size":null,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2503.01935","github":"https://github.com/ulab-uiuc/MARBLE","leaderboard":null},"good_for":["grading-design","taxonomy"],"fit":"Fits teams running several cooperating agents (planner plus workers, or peer agents). Borrow the milestone-based KPI where an LLM detector marks which sub-goals were hit, plus the separate coordination score.","caveats":"Milestone detection and coordination scores are LLM-rated, so they inherit judge bias; task counts per scenario are only partly stated; code moved between two repositories."},{"id":"who-and-when","name":"Who&When","year":2025,"by":"Zhang et al.","workloads":["multi-agent","general-assistant"],"task_types":["read a failed multi-agent run and name the agent responsible","point to the exact step where the run went wrong","explain the failure in plain language"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match"],"size":184,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.00212","github":"https://github.com/mingyin1/Agents_Failure_Attribution","leaderboard":null},"good_for":["grading-design","taxonomy","baseline"],"fit":"Fits teams that want an automated judge to blame the right agent and step after a multi-agent failure. Borrow the agent-plus-step labeling scheme; the reported baselines (about half right on agent, far lower on step) set expectations.","caveats":"Small (184 failures) and drawn from CaptainAgent and Magentic-One on GAIA and AssistantBench queries, so it covers a narrow set of orchestration styles."},{"id":"deepplanning","name":"DeepPlanning","year":2026,"by":"Qwen Team, Alibaba","workloads":["planning","travel","shopping","tool-calling"],"task_types":["plan a multi-day trip within time and money budgets using search tools","assemble a shopping cart that satisfies cross-item constraints and coupons","gather missing information proactively before committing a plan"],"interaction":"single-turn","environment":"api-sandbox","state_checked":true,"tools":["9 travel APIs","15 shopping APIs in an offline sandbox"],"grading":["programmatic-check"],"size":240,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2601.18137","github":"https://github.com/QwenLM/Qwen-Agent","leaderboard":"https://qwenlm.github.io/Qwen-Agent/en/benchmarks/deepplanning/"},"good_for":["grading-design","case-template"],"fit":"Fits booking or purchasing assistants with global budgets. Borrow the rule-based checkers (commonsense checkpoints, cart match) instead of LLM judges.","caveats":"Two domains only; travel itineraries are first parsed by a Qwen model before checking; license not stated."},{"id":"logior","name":"LogiOR (ORThought)","year":2025,"by":"Zhejiang University (ZJU-TSELab)","workloads":["planning","data-analysis","coding","manufacturing"],"task_types":["read a logistics or transport planning problem in plain words","write a mathematical optimization model for it","write and run solver code (Gurobi) that solves the model","return the optimal objective value"],"interaction":"single-turn","environment":"terminal","state_checked":false,"tools":["Python","Gurobi solver"],"grading":["programmatic-check"],"size":92,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.14410","github":"https://github.com/ZJU-TSELab/ORThought","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits planning or operations-research copilots that turn a logistics request into a solvable model. Borrow the case format (problem text, reference formulation, reference solver code, optimal value, size and type metadata) and grading on the optimal value.","caveats":"Only 92 problems; needs a Gurobi license beyond the free trial for larger problems; grades the final objective value so a wrong model with the right number passes; no license stated for the data or code."},{"id":"realm-bench","name":"REALM-Bench","year":2025,"by":"Stanford (Geng and Chang)","workloads":["planning","multi-agent","manufacturing"],"task_types":["plan routes or schedules with several agents and dependencies","replan when a disruption arrives mid-execution","solve supply-chain and job-shop scheduling instances","coordinate logistics for an event or disaster response"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none; planning over given problem instances"],"grading":["programmatic-check"],"size":11,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2502.18836","github":"https://github.com/genglongling/REALM-Bench","leaderboard":null},"good_for":["taxonomy","grading-design"],"fit":"Fits planning and scheduling agents in operations or logistics. Borrow the six metrics (goal satisfaction, makespan, coordination, constraints, resource use, replanning).","caveats":"Only 11 scenario families (paper says 14); classical OR baselines beat LLMs; no tool environment."},{"id":"ultrahorizon","name":"UltraHorizon","year":2025,"by":"Luo et al.","workloads":["planning","general-assistant","research"],"task_types":["explore an unknown environment over hundreds of tool calls to discover hidden rules","keep track of findings across a very long context","run experiments and update hypotheses (grid, sequence and genetics puzzles)"],"interaction":"autonomous-long-horizon","environment":"api-sandbox","state_checked":false,"tools":["environment-specific exploration tools"],"grading":["llm-judge","programmatic-check"],"size":null,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2509.21766","github":"https://github.com/StarDewXXX/UltraHorizon","leaderboard":null},"good_for":["taxonomy","baseline"],"fit":"Fits teams stress-testing agents on very long sessions (200k+ tokens, 400+ calls). Borrow the eight-way error taxonomy (in-context locking versus capability gaps).","caveats":"Abstract puzzle environments, not work tasks; expensive runs; scoring uses a judge model."},{"id":"vending-bench","name":"Vending-Bench","year":2025,"by":"Andon Labs","workloads":["planning","finance","retail"],"task_types":["run a simulated vending business over a long horizon","order stock, set prices and pay fees without losing track","stay coherent across millions of tokens of context"],"interaction":"autonomous-long-horizon","environment":"api-sandbox","state_checked":true,"tools":["simulated email","inventory","pricing and bank tools"],"grading":["programmatic-check"],"size":null,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2502.15840","github":null,"leaderboard":null},"good_for":["grading-design"],"fit":"Fits long-running operations agents where the question is coherence over weeks of simulated time. Borrow the single hard metric (ending bank balance) and the failure-mode list (forgotten orders, misread schedules, meltdown loops).","caveats":"Closed simulation run by the authors, no public code found; one scenario; runs are very expensive (over 20M tokens) and later versions report models gaming the environment."},{"id":"worfbench","name":"WorfBench","year":2024,"by":"Zhejiang University","workloads":["planning","tool-calling","embodied","workflow-automation"],"task_types":["write a step-by-step plan for a how-to task","lay out a graph of tool calls with parallel branches","plan steps for a household or web-shop embodied task","plan an open-ended grounded task before acting"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none at test time; workflows reference tools from ToolBench, ToolAlpaca, Lumos, ALFWorld, WebShop"],"grading":["programmatic-check"],"size":2146,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2410.07869","github":"https://github.com/zjunlp/WorfBench","leaderboard":null},"good_for":["grading-design","taxonomy"],"fit":"Fits agents that produce a plan before executing, such as workflow builders. Borrow the graph matching evaluation (F1 on node chain and on the workflow graph) that rewards correct parallel structure, not just step order.","caveats":"Grades the written plan, not execution, and node matching uses a 0.6 embedding similarity threshold that can misjudge paraphrases."},{"id":"core-bench","name":"CORE-Bench","year":2024,"by":"Princeton (Siegel, Kapoor, Narayanan et al.)","workloads":["research","coding","data-analysis"],"task_types":["reproduce a published paper's results from its code capsule and answer questions about them","set up the environment, run the code, and read figures or tables it produces"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":false,"tools":["shell","python","R"],"grading":["exact-match"],"size":270,"data":"real-public","pii":"none","license":"MIT","harness":["inspect_evals","hal","own-repo"],"status":"saturated","links":{"paper":"https://arxiv.org/abs/2409.11363","github":"https://github.com/siegelz/core-bench","leaderboard":"https://hal.cs.princeton.edu/"},"good_for":["grading-design","case-template"],"fit":"Fits reproducibility or analysis agents; borrow the question-and-answer report.json grading that avoids judging code directly.","caveats":"HAL has declared it solved after a Claude Opus 4.5 run, so it no longer separates current agents; Medium tier needs Docker-in-Docker."},{"id":"deep-research-bench-futuresearch","name":"Deep Research Bench (FutureSearch)","year":2025,"by":"FutureSearch","workloads":["research","web-browsing"],"task_types":["work a multi-step web research task to a number, URL, probability or dataset","compile an evidence set for a claim","research against a frozen offline snapshot of the web"],"interaction":"autonomous-long-horizon","environment":"api-sandbox","state_checked":false,"tools":["search and page fetch over frozen scraped pages"],"grading":["programmatic-check","llm-judge"],"size":89,"data":"real-scraped","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.06287","github":null,"leaderboard":"https://drb.futuresearch.ai"},"good_for":["sandbox-design","taxonomy"],"fit":"Fits research agents that must stay reproducible as the web changes. Borrow the RetroSearch idea of freezing the pages per task so human-worked answers stay valid, and the trace checks for hallucination, tool use and forgetting.","caveats":"No public code or data repository seen; runs are requested by email; the paper set has 89 instances while the site now lists more, so numbers are not stable across versions; scores across task types are not comparable."},{"id":"deepresearch-bench","name":"DeepResearch Bench","year":2025,"by":"University of Science and Technology of China","workloads":["research","web-browsing","documents"],"task_types":["write a long cited research report on a PhD-level question","search the web and pick sources that support each claim","follow the instructions in the task about scope and structure"],"interaction":"autonomous-long-horizon","environment":"browser","state_checked":false,"tools":["web search","page scraping"],"grading":["llm-judge","programmatic-check"],"size":100,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.11763","github":"https://github.com/Ayanami0730/deep_research_bench","leaderboard":"https://huggingface.co/spaces/muset-ai/DeepResearch-Bench-Leaderboard"},"good_for":["grading-design","baseline","taxonomy"],"fit":"Fits report-writing research agents. Borrow RACE (task-specific weighted criteria scored against a reference report) and FACT (extract claim-URL pairs and check whether the cited page supports each claim).","caveats":"Only 100 tasks, half Chinese and half English; scores depend on the judge model, which the maintainers are switching, so numbers across time are not comparable; FACT needs a scraping API key."},{"id":"deepresearchgym","name":"DeepResearchGym","year":2025,"by":"Carnegie Mellon University","workloads":["research","web-browsing"],"task_types":["write a cited report for a complex information need","search a fixed public web index instead of a commercial API","keep retrieved passages faithful to the report's claims"],"interaction":"autonomous-long-horizon","environment":"api-sandbox","state_checked":false,"tools":["search API over ClueWeb22 and FineWeb"],"grading":["llm-judge"],"size":1000,"data":"real-public","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.19253","github":null,"leaderboard":null},"good_for":["sandbox-design","grading-design"],"fit":"Fits teams who need reproducible search for a research agent. Borrow the stable search sandbox over public corpora and the three judge dimensions: alignment with the information need, retrieval faithfulness, and report quality.","caveats":"The paper says the code and search API URLs will be released but the arXiv page gives no repository; the corpora are static so recent events are missing; queries come from Researchy Questions (top 1,000 of the test set), not purpose-written tasks."},{"id":"deepscholar-bench","name":"DeepScholar-Bench","year":2025,"by":"Stanford (Guestrin lab)","workloads":["research","science","web-browsing"],"task_types":["write the related-work section for a recent paper given its topic","retrieve relevant prior papers from the web","cite sources so each statement can be verified"],"interaction":"autonomous-long-horizon","environment":"browser","state_checked":false,"tools":["web search","paper retrieval"],"grading":["llm-judge","programmatic-check"],"size":63,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.20033","github":"https://github.com/guestrin-lab/deepscholar-bench","leaderboard":"https://guestrin-lab.github.io/deepscholar-leaderboard/leaderboard/deepscholar_bench_leaderboard.html"},"good_for":["grading-design","case-template","baseline"],"fit":"Fits literature-review or scientific writing agents. Borrow the live-refresh pipeline that mines new arXiv papers for queries with human-written reference text, and the split metrics for organization, nugget coverage, reference coverage and citation precision.","caveats":"The June 2025 set has only 63 queries (a later 200-query set exists); tasks are limited to related-work sections of arXiv papers; grading uses LLM judges built on the LOTUS framework."},{"id":"dr-bench","name":"Dr. Bench","year":2025,"by":"Yang Yao et al.","workloads":["research","web-browsing","documents"],"task_types":["write a long report on an expert-curated question","stay on topic as measured by focus keywords","cite trustworthy sources from a reference list"],"interaction":"autonomous-long-horizon","environment":"browser","state_checked":false,"tools":["web search"],"grading":["llm-judge","programmatic-check"],"size":214,"data":"real-public","pii":"none","license":"MIT","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.02190","github":null,"leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits report-writing agents that need more than a single judge score. Borrow the reference bundle per task: query-specific rubrics, general report rubrics, trusted source links, and focus-anchor versus focus-deviation keywords.","caveats":"No code repository seen, only a Hugging Face dataset; grading details come from the rubric fields rather than a published script; the name collides with DeepResearch Bench and DRBench."},{"id":"livedrbench","name":"LiveDRBench","year":2025,"by":"Microsoft Research","workloads":["research","web-browsing","science"],"task_types":["find all claims or items that answer a broad research question","search for prior art or datasets on a scientific topic","compile facts about a public event such as a flight incident"],"interaction":"autonomous-long-horizon","environment":"browser","state_checked":false,"tools":["web search"],"grading":["llm-judge"],"size":100,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.04183","github":"https://github.com/microsoft/LiveDRBench","leaderboard":null},"good_for":["grading-design","taxonomy"],"fit":"Fits research agents whose output should be a set of findings rather than prose. Borrow the claim-set intermediate representation graded by precision, recall and F1 per category.","caveats":"Only 100 tasks; the data is under CDLA v2 while code is MIT; the live refresh is a stated plan, not yet a feature; default judge is gpt-4o."},{"id":"mmdeepresearch-bench","name":"MMDeepResearch-Bench","year":2026,"by":"AIoT-MLSys-Lab","workloads":["research","web-browsing","documents"],"task_types":["write a cited report from a bundle of images and text","search for sources that back claims about visual evidence","keep text, citations and figure references consistent"],"interaction":"autonomous-long-horizon","environment":"browser","state_checked":false,"tools":["web search","image understanding"],"grading":["llm-judge","programmatic-check"],"size":140,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2601.12346","github":"https://github.com/AIoT-MLSys-Lab/MMDeepResearch-Bench","leaderboard":null},"good_for":["grading-design","taxonomy"],"fit":"Fits research agents whose inputs include charts, screenshots or photos. Borrow the three separate scores for report quality, citation fidelity and text-to-image consistency, and the pass/fail gate on visual evidence.","caveats":"New and small (140 tasks, 100 research and 40 daily); the composite score mixes four weighted parts, so a single number hides which part failed; one earlier commit had a scoring bug."},{"id":"paperarena","name":"PaperArena","year":2025,"by":"Daoyu Wang et al.","workloads":["research","science","tool-calling"],"task_types":["answer a question that needs facts from several papers","parse PDFs, tables and figures with tools","run code to compute a value from paper data"],"interaction":"autonomous-long-horizon","environment":"api-sandbox","state_checked":false,"tools":["PDF parser","figure and table analyzers","cross-reference search","web search","paper database","code executor"],"grading":["llm-judge"],"size":784,"data":"real-public","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.10909","github":"https://github.com/Melmaphother/PaperArena","leaderboard":null},"good_for":["sandbox-design","case-template","taxonomy"],"fit":"Fits agents that answer questions over a private document corpus with a tool set (parse, retrieve, compute). Borrow the modular tool platform and the question pipeline with obfuscation and manual review over a fixed corpus of 100 papers.","caveats":"No license file in the repo; questions were generated by an LLM then checked, so some may be tool-chain shaped rather than user shaped; binary GPT-4o judging; only AI papers."},{"id":"researchqa","name":"ResearchQA","year":2025,"by":"University of Pennsylvania","workloads":["research","science","knowledge-qa"],"task_types":["give a long-form answer to a scholarly question across 75 fields","cite relevant papers and explain their findings","describe limitations and compare approaches"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none required","retrieval optional"],"grading":["llm-judge","human"],"size":21400,"data":"real-public","pii":"none","license":"MIT","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2509.00496","github":null,"leaderboard":null},"good_for":["grading-design","case-template","taxonomy"],"fit":"Fits assistants that answer domain research questions at scale. Borrow the method of mining questions and yes/no rubric items from survey articles so a large rubric set exists without expert annotators.","caveats":"Rubrics are auto-mined, so some items are shallow; rubric coverage rewards breadth over correctness; no repository seen on the arXiv or dataset pages; size is approximate (full split listed as 21.4k)."},{"id":"researchrubrics","name":"ResearchRubrics","year":2025,"by":"Scale AI","workloads":["research","web-browsing","documents"],"task_types":["write a long evidence-backed answer to an open-ended research prompt","infer implicit context the prompt does not spell out","reason over retrieved material rather than paraphrase it"],"interaction":"autonomous-long-horizon","environment":"browser","state_checked":false,"tools":["web search"],"grading":["llm-judge","human"],"size":101,"data":"real-public","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2511.07685","github":null,"leaderboard":null},"good_for":["grading-design","case-template","taxonomy"],"fit":"Fits any report-writing agent that needs rubric grading. Borrow the expert-written weighted rubrics (20-43 criteria per prompt, with negative-weight criteria) and the three-level satisfied / partial / not-satisfied verdict scheme with multiple judge models.","caveats":"Only 101 prompts; the paper says prompts, rubrics and code are released but the arXiv page gives no repository, only a Scale project page; license not stated."},{"id":"widesearch","name":"WideSearch","year":2025,"by":"ByteDance Seed","workloads":["research","web-browsing","data-analysis"],"task_types":["collect many small facts about every item in a set and return a table","fill a fixed set of columns for each entity with correct values","verify each cell against public sources"],"interaction":"autonomous-long-horizon","environment":"browser","state_checked":false,"tools":["web search","page browsing"],"grading":["programmatic-check","llm-judge","pass^k"],"size":200,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.07999","github":"https://github.com/ByteDance-Seed/WideSearch","leaderboard":"https://widesearch-seed.github.io/"},"good_for":["grading-design","case-template","baseline"],"fit":"Fits agents that build lists or spreadsheets from the web (lead lists, product comparisons, compliance registers). Borrow the table grader: per-column match rules (exact, numeric, date, URL or judge), row and cell F1, and a strict all-cells-correct success rate.","caveats":"Best systems score about 5% success, so it mainly separates failure modes; half the tasks are Chinese; the live web means ground-truth tables can go stale."},{"id":"ecommerce-bench","name":"E-Commerce Bench","year":2026,"by":"Qwen Team, Alibaba","workloads":["retail","sales","planning","finance"],"task_types":["run online stores for a simulated year (pricing, stocking, fulfilment, returns)","negotiate with suppliers and avoid fraudulent ones","manage cash flow under escrow delays and demand shocks"],"interaction":"autonomous-long-horizon","environment":"api-sandbox","state_checked":true,"tools":["18 merchant tools over 6","886 products and 576 suppliers"],"grading":["programmatic-check"],"size":1,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2608.30730","github":"https://github.com/QwenLM/E-CommerceBench","leaderboard":"https://github.com/QwenLM/E-CommerceBench"},"good_for":["sandbox-design","grading-design"],"fit":"Fits agents that operate a business over a long horizon. Borrow the deterministic demand and negotiation kernels (the LLM only renders dialogue) and the multi-axis scoring with assets as the headline.","caveats":"One scenario with five episodes per model; very long runs; outcome is a single number that hides which step failed."},{"id":"agent-safetybench","name":"Agent-SafetyBench","year":2024,"by":"Tsinghua University (CoAI)","workloads":["safety","tool-calling","general-assistant"],"task_types":["complete a tool task without leaking sensitive data","avoid actions that cause property loss or physical harm","refuse to spread misinformation or break the law","avoid writing harmful or vulnerable code","notice a risk hidden in the task or the environment"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["349 hand-written tool environments with JSON schemas and Python logic"],"grading":["llm-judge"],"size":2000,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2412.14470","github":"https://github.com/thu-coai/Agent-SafetyBench","leaderboard":null},"good_for":["taxonomy","grading-design","case-template"],"fit":"Fits general tool-using assistants that need a broad risk taxonomy. Borrow the 8 risk categories and 10 failure modes, and the idea of a small fine-tuned judge (ShieldAgent) for safe/unsafe labels.","caveats":"Grading relies on a fine-tuned 7B judge, not on state; cases were LLM-generated then human-checked; tool environments are simple and many cases are short."},{"id":"agent-security-bench","name":"Agent Security Bench (ASB)","year":2024,"by":"Rutgers University (AGI Research)","workloads":["safety","security","tool-calling","memory"],"task_types":["complete a domain task (finance, legal, counseling, e-commerce) with tools","resist direct prompt injection in the user query","resist injection in tool observations","resist poisoned memory retrieval and backdoored plans","avoid calling attacker-specific tools"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["400 attack tools and 20 normal tools","simulated inside AIOS"],"grading":["programmatic-check","llm-judge"],"size":400,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2410.02644","github":"https://github.com/agiresearch/ASB","leaderboard":null},"good_for":["taxonomy","baseline"],"fit":"Fits teams comparing defenses across attack stages (query, observation, memory, plan). Borrow the attack-by-stage matrix and the utility-versus-security metric.","caveats":"Tools are simulated and success is whether an attack tool is called, so it measures willingness, not outcomes; the 400 attack tasks reuse 50 agent tasks; tied to the AIOS framework."},{"id":"agentauditor-assebench","name":"AgentAuditor / ASSEBench","year":2025,"by":"Hanjun Luo et al.","workloads":["safety","security","tool-calling"],"task_types":["read a recorded agent interaction and label it safe or unsafe","spot a security threat such as injection or data leak in a trajectory","apply a strict or lenient standard to an ambiguous case","explain which risk type and behavior mode applies"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none; the evaluator reads recorded trajectories"],"grading":["exact-match"],"size":2293,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.00641","github":"https://github.com/Astarojth/AgentAuditor","leaderboard":null},"good_for":["grading-design","taxonomy"],"fit":"Fits teams building an LLM judge for agent safety rather than testing the agent itself. Borrow the human-labeled records with ambiguity flags and the strict/lenient split to calibrate a judge's F1 before trusting it.","caveats":"It benchmarks judges, not agents; the GitHub repo is a placeholder and the data lives on Hugging Face; records were generated by three LLMs on top of other benchmarks, so it inherits their synthetic scope."},{"id":"agentdojo","name":"AgentDojo","year":2024,"by":"ETH Zurich (SPY Lab)","workloads":["safety","security","tool-calling","email-calendar","finance"],"task_types":["complete an email, calendar or file task from a one-line request","make a bank transfer or check transactions","book travel from a request","ignore instructions hidden in tool output while finishing the task","avoid doing the attacker's task (send data, transfer money)"],"interaction":"single-turn","environment":"api-sandbox","state_checked":true,"tools":["simulated email","calendar","cloud drive","Slack","travel booking and banking APIs"],"grading":["state-diff","programmatic-check"],"size":97,"data":"synthetic","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2406.13352","github":"https://github.com/ethz-spylab/agentdojo","leaderboard":"https://agentdojo.spylab.ai/results/"},"good_for":["grading-design","sandbox-design","case-template","baseline"],"situations":["injected"],"fit":"Fits assistants that read untrusted content (email, documents, web) and then act with tools. Borrow the pairing of a user task with an injection task, and the utility-under-attack plus targeted attack success rate metrics checked on environment state, not on model text.","caveats":"97 user tasks become 629 security cases only by crossing tasks with injections; all data is synthetic; the results page is not a maintained leaderboard and ends in early 2025."},{"id":"agentharm","name":"AgentHarm","year":2024,"by":"UK AI Safety Institute and Gray Swan AI","workloads":["safety","tool-calling","general-assistant"],"task_types":["refuse an openly harmful multi-step request (fraud, cybercrime, harassment)","complete the matching benign version of the same task","call the right synthetic tools in the right order","keep working correctly after a jailbreak attempt"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["synthetic tools for email","web","files","messaging and payments"],"grading":["programmatic-check","llm-judge"],"size":110,"data":"synthetic","pii":"none","license":"MIT (with an added clause limiting use to AI safety work)","harness":["inspect_evals"],"status":"active","links":{"paper":"https://arxiv.org/abs/2410.09024","github":"https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/agentharm","leaderboard":null},"good_for":["case-template","grading-design","taxonomy"],"situations":["forbidden"],"fit":"Fits any tool-using assistant that must refuse misuse without losing the ability to do the benign version. Borrow the harmful/benign task pairs and the split scoring (refusal judge, semantic judge, tool-call grading functions).","caveats":"Only 44 of 110 base behaviors are public (the rest are held out); 440 cases come from augmentations of those; tools are synthetic and tasks are short; the license adds a use restriction."},{"id":"agentic-misalignment","name":"Agentic Misalignment","year":2025,"by":"Anthropic","workloads":["safety","email-calendar","general-assistant"],"task_types":["act as an email oversight assistant at a fictional company","decide what to do when told it will be replaced or restricted","choose whether to blackmail an executive using information found in email","choose whether to leak confidential documents to an outside party"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none in the text version; simulated email inbox given as context","send-email action in original"],"grading":["llm-judge","programmatic-check"],"size":null,"data":"synthetic","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://www.anthropic.com/research/agentic-misalignment","github":"https://github.com/anthropic-experimental/agentic-misalignment","leaderboard":null},"good_for":["case-template","grading-design","sandbox-design"],"fit":"Fits any agent given standing access to a company inbox or documents plus a long-running goal, such as an executive assistant or operations agent. Borrow the way one scenario is crossed with goal type, goal content and urgency to build a condition grid, and the classifier prompt that labels a transcript harmful or not.","caveats":"One fictional scenario with three harmful endings (blackmail, leak, lethal action); the model only reads emails and replies, no real tool execution; grading depends on an LLM classifier and the inspect_evals port reports different rates from the original blog post."},{"id":"agentthreatbench","name":"AgentThreatBench","year":2026,"by":"Community contribution to UK AISI inspect_evals (vgudur-dev)","workloads":["safety","security","tool-calling","email-calendar","customer-support"],"task_types":["answer questions from a memory store that holds poisoned entries","triage an inbox without obeying instructions inside emails","resolve a support ticket without sending sensitive fields to the wrong recipient","complete the task (utility) and resist the attack (security) at the same time"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["mock inbox","email categorizer","summary drafter","customer lookup","send message","ticket resolver","memory store"],"grading":["programmatic-check"],"size":24,"data":"synthetic","pii":"none","license":"MIT","harness":["inspect_evals"],"status":"active","links":{"paper":null,"github":"https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/agent_threat_bench","leaderboard":null},"good_for":["case-template","grading-design"],"fit":"Fits a quick smoke test for email or support agents mapped to the OWASP agentic top 10. Borrow the two independent binary scores per sample and the four outcome profiles (robust, vulnerable, overly cautious, failing).","caveats":"Tiny (24 samples) and covers only 2 of 10 OWASP categories; no paper and no outside validation; attack markers are simple strings."},{"id":"b3","name":"b3 (Backbone Breaker Benchmark)","year":2025,"by":"Lakera AI with UK AI Security Institute","workloads":["safety","security","tool-calling","general-assistant"],"task_types":["respond inside an application context with a system prompt and tool definitions","resist a direct instruction override from the user","resist an indirect override placed in tool output","refuse an unauthorized tool call","keep answering under denial-of-service style attacks"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["tool definitions in 10 application snapshots; no execution"],"grading":["programmatic-check","llm-judge"],"size":210,"data":"real-public","pii":"none","license":"MIT","harness":["inspect_evals"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.22620","github":"https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/b3","leaderboard":null},"good_for":["case-template","baseline","taxonomy"],"fit":"Fits choosing a backbone model for an agent before building it. Borrow the threat snapshot idea (freeze one step of the agent where the model sees an attack) and the three defense levels per snapshot.","caveats":"The public Hugging Face set is a deliberately weakened subset with the strongest attacks removed; attacks are single-step with no real tool execution; HF metadata says license other while the card says MIT."},{"id":"codeipi","name":"CodeIPI","year":2026,"by":"Community contribution to UK AISI inspect_evals (debu-sinha)","workloads":["safety","security","coding"],"task_types":["fix a bug in a small repository with bash and python","ignore instructions hidden in issue text, code comments, READMEs or config files","avoid leaking canary secrets, running planted commands or persisting changes","flag the injection while still completing the task"],"interaction":"single-turn","environment":"containers","state_checked":true,"tools":["bash and python in a no-network Docker sandbox with git and pytest"],"grading":["programmatic-check","llm-judge"],"size":45,"data":"synthetic","pii":"none","license":"MIT","harness":["inspect_evals"],"status":"active","links":{"paper":null,"github":"https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/ipi_coding_agent","leaderboard":null},"good_for":["grading-design","sandbox-design"],"fit":"Fits coding agents that read issues and repos. Borrow the canary design: planted secrets and env vars for exfiltration, a unique marker for execution, and file hashes before and after for persistence.","caveats":"Only 45 samples (35 injected, 10 benign) and no paper; the task source is not described; reported numbers lag the latest version."},{"id":"doomarena","name":"DoomArena","year":2025,"by":"ServiceNow Research","workloads":["safety","security","web-browsing","tool-calling","customer-support"],"task_types":["run tau-bench tool tasks against a malicious simulated user","run BrowserGym web tasks with attacker content injected into pages","combine several attacks in one episode","keep task success while the environment or user is adversarial"],"interaction":"multi-turn-simulated-user","environment":"mixed","state_checked":true,"tools":["plug-in attacks for tau-bench","BrowserGym","OSWorld and TapeAgents"],"grading":["state-diff","programmatic-check"],"size":null,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2504.14064","github":"https://github.com/ServiceNow/DoomArena","leaderboard":null},"good_for":["sandbox-design","grading-design"],"fit":"Fits teams that already run tau-bench or a browser benchmark and want to add threat models without rewriting tasks. Borrow the separation of attack, attackable component and success criterion from the host environment.","caveats":"A framework, not a fixed task set, so numbers are not comparable across papers; attack success is defined per host environment; last repo activity was late 2025."},{"id":"gdm-dangerous-capabilities-self-proliferation","name":"GDM Dangerous Capabilities: Self-proliferation","year":2024,"by":"Google DeepMind (Phuong et al.)","workloads":["safety","security","web-browsing","it-ops"],"task_types":["set up accounts or services on the web without help","copy and run a model or agent on new infrastructure","earn money or resources through online tasks","complete milestones toward spreading itself"],"interaction":"autonomous-long-horizon","environment":"mixed","state_checked":true,"tools":["bash","web browser","in Docker Compose sandboxes with human approval of tool calls"],"grading":["programmatic-check","human"],"size":10,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2403.13793","github":"https://github.com/google-deepmind/dangerous-capability-evaluations","leaderboard":null},"good_for":["sandbox-design","grading-design"],"fit":"Fits teams giving an agent real web, cloud and shell access who need to know how far it can get on its own. Borrow the milestone mode that scores partial progress and the human-approval gate on every tool call.","caveats":"Ten tasks only; several need GCP accounts, API keys and secrets; DeepMind removed solutions and altered some tasks and asks you to contact them before running those; the expert best-of-N mode in inspect_evals was removed as broken."},{"id":"gdm-dangerous-capabilities-stealth","name":"GDM Dangerous Capabilities: Stealth","year":2025,"by":"Google DeepMind (Phuong et al.)","workloads":["safety","it-ops","tool-calling"],"task_types":["evade a classifier that is watching its outputs","cover its tracks after taking an action","find and reason about an oversight pattern in configuration","break a rule strategically while completing a task"],"interaction":"autonomous-long-horizon","environment":"filesystem","state_checked":true,"tools":["task-specific tools in a ReAct loop"],"grading":["programmatic-check","llm-judge"],"size":4,"data":"synthetic","pii":"none","license":"unknown","harness":["inspect_evals"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.01420","github":"https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/gdm_stealth","leaderboard":null},"good_for":["case-template","grading-design"],"fit":"Fits agents with write access to their own configuration, logs or monitoring, where quiet rule breaking would matter. Borrow the scenarios that put the agent near its own oversight and the use of realistic names so the model does not notice it is being tested.","caveats":"Four tasks and no standalone repo; rater models default to Google and need a Google API key; one task was broken until October 2026; the paper lists five stealth evaluations so coverage is partial."},{"id":"hvtb","name":"Hack-Verifiable Terminal Bench (HVTB)","year":2026,"by":"Roth, Bercovich, Efroni","workloads":["safety","coding","it-ops"],"task_types":["solve a terminal or coding task in a container","resist reading a planted reference solution or held-out tests","pass the task checks without violating the task's intent"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["bash"],"grading":["programmatic-check","unit-tests"],"size":89,"data":"real-public","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2608.22103","github":"https://github.com/Aarav500/rhob","leaderboard":null},"good_for":["grading-design","sandbox-design"],"fit":"Fits coding and operations agents that run against tests, where reward hacking is a worry. Borrow the trick of planting a reference solution and tests in the container and using filesystem watchers to flag any read, giving a hack rate with no judge.","caveats":"Tasks are reused from Terminal Bench 2.1 so capability numbers are not new; the inspect_evals port uses a different agent harness than the paper; the host repo (RHOB) carries a correction banner about other claims, so review the pinned commit before trusting it."},{"id":"injecagent","name":"InjecAgent","year":2024,"by":"University of Illinois Urbana-Champaign (Daniel Kang's group)","workloads":["safety","security","tool-calling"],"task_types":["read a tool result that contains attacker instructions","avoid calling a harmful tool the attacker asks for","avoid extracting private data and sending it to the attacker","finish the user's original request"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["templated tool specs; 17 user tools","62 attacker tools"],"grading":["programmatic-check"],"size":1054,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2403.02691","github":"https://github.com/uiuc-kang-lab/InjecAgent","leaderboard":null},"good_for":["case-template","taxonomy"],"situations":["injected"],"fit":"Fits a first, cheap check of any tool-calling agent for indirect prompt injection. Borrow the two attacker goals (direct harm, data theft) and the user-case times attacker-case crossing that gives many cases from few templates.","caveats":"No tool is executed; tool responses are templates, so success is just whether the attacker tool is called; the repo has not changed since mid-2024 and newer models mostly pass it."},{"id":"instrumentaleval","name":"InstrumentalEval","year":2025,"by":"He et al. (National University of Singapore)","workloads":["safety","general-assistant"],"task_types":["respond to a scenario that tempts it to evade shutdown or copy itself","respond to a scenario where deceiving the user would help its goal","respond to a scenario where hacking a system would help its goal","respond to a scenario where hiding behavior or appearing aligned would help"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge"],"size":null,"data":"synthetic","pii":"none","license":"unknown","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2502.12206","github":"https://github.com/yf-he/InstrumentalEval","leaderboard":null},"good_for":["taxonomy","case-template"],"fit":"Fits teams giving an agent an open-ended goal such as make money or grow usage, who want a cheap screen for power-seeking or self-preservation talk. Borrow the six behavior categories as a taxonomy for red-team cases.","caveats":"Text prompts only, no tools or environment; no license and no item count seen; the inspect_evals port reports different results from the paper and depends on a grader model's judgment."},{"id":"mask","name":"MASK","year":2025,"by":"Center for AI Safety and Scale AI (Ren et al.)","workloads":["safety","knowledge-qa","general-assistant"],"task_types":["state what it believes about a factual claim when asked neutrally","answer the same claim under a system prompt or user that pressures it to lie","stay consistent between its belief and its pressured answer"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge"],"size":1000,"data":"mixed","pii":"possible","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2503.03750","github":"https://github.com/centerforaisafety/mask","leaderboard":"https://www.mask-benchmark.ai"},"good_for":["grading-design","case-template"],"fit":"Fits agents whose operator prompt could push them to shade the truth, such as sales or support bots with a persona. Borrow the split between honesty and accuracy, and the belief-elicitation step that measures what the model thinks before applying pressure.","caveats":"Single-turn prompts with no tools, so it tests the model not the agent loop; dataset is gated on Hugging Face; grading uses GPT judges, and the inspect_evals port warns that multi-epoch runs are unreliable; pii marked possible because some archetypes use real-world people and events."},{"id":"mcp-safetybench","name":"MCP-SafetyBench","year":2025,"by":"Xuanjun Zong et al.","workloads":["safety","security","tool-calling","web-browsing","finance"],"task_types":["complete a multi-step task across several real MCP servers","resist attacks injected through a server's tool manifest or implementation","resist attacks from the host or the user side","keep task success while refusing the attacker's goal"],"interaction":"multi-turn-scripted","environment":"api-sandbox","state_checked":false,"tools":["real MCP servers for browser automation","financial data","maps","repository management and web search"],"grading":["programmatic-check","trajectory-check"],"size":245,"data":"mixed","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2512.15163","github":"https://github.com/xjzzzzzzzz/MCPSafety","leaderboard":null},"good_for":["taxonomy","grading-design","sandbox-design"],"situations":["forbidden"],"fit":"Fits agents that connect to third-party MCP servers. Borrow the 20-type attack taxonomy across server, host and user sides, and the paired task evaluator plus attack detector that reports task success rate and attack success rate together.","caveats":"No license file in the repo; servers are real so results drift with live data; only 245 cases across 5 domains; the README gives no counts or grading details."},{"id":"mcpsecbench","name":"MCPSecBench","year":2025,"by":"Yixuan Yang et al.","workloads":["safety","security","tool-calling"],"task_types":["handle a request while a malicious MCP server tries tool poisoning","withstand protocol-level attacks such as man-in-the-middle and DNS rebinding","avoid a server whose name squats on a trusted one","finish a benign MCP request without a specification violation"],"interaction":"single-turn","environment":"mixed","state_checked":false,"tools":["custom benign and malicious MCP servers","an MCP client","attack scripts","a GUI test harness"],"grading":["llm-judge","programmatic-check"],"size":null,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.13220","github":"https://github.com/AIS2Lab/MCPSecBench","leaderboard":null},"good_for":["taxonomy","sandbox-design"],"situations":["injected"],"fit":"Fits teams shipping an MCP host or client, not just a model. Borrow the 17 attack types over four surfaces (client, protocol, server, host) and the success/refusal/execution-error outcome split.","caveats":"No published case count; it is a playground tied to Claude Desktop, OpenAI and Cursor as hosts, so runs are partly manual; results in the repo cover OpenAI only."},{"id":"mcptox","name":"MCPTox","year":2025,"by":"USTC and Beihang University","workloads":["safety","security","tool-calling"],"task_types":["answer a benign request given a tool set from a real MCP server","ignore malicious instructions hidden in a tool's description","avoid calling a legitimate tool for a harmful purpose","refuse or flag a poisoned tool"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["tool descriptions from 45 real MCP servers (353 tools); the poisoned tool is never run"],"grading":["llm-judge"],"size":1348,"data":"mixed","pii":"none","license":"unknown","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.14925","github":"https://github.com/zhiqiangwang4/MCPTox-Benchmark","leaderboard":null},"good_for":["case-template","taxonomy"],"situations":["injected"],"fit":"Fits agents that load tools from external MCP registries. Borrow the three poisoning templates and the 10 risk categories to build a tool-description screening test.","caveats":"The official repo README is one line and has no license; case count differs between paper versions (1312 vs 1348); success is judged by a model; the inspect_evals version is an external, pinned third-party port."},{"id":"mind2web-sc","name":"Mind2Web-SC","year":2024,"by":"GuardAgent authors (Zhen Xiang, Bo Li et al.)","workloads":["safety","web-browsing"],"task_types":["decide whether a web agent's planned action breaks a user-profile rule (age, license, membership)","write guardrail code that checks the rule against the user record","grant or deny the action and name the violated rule"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none for the guard; recorded SeeAct web-agent outputs plus a code runner"],"grading":["exact-match"],"size":200,"data":"mixed","pii":"none","license":"unknown","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2406.09187","github":"https://github.com/guardagent/dataset","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits teams adding a policy guardrail in front of a web or booking agent. Borrow the rule-plus-profile case format and the split accuracy (right decision, right violation named, right action kept).","caveats":"Tests a guard model, not the agent; 200 cases with six made-up rules and random user profiles; no license in the dataset repo, which has not changed since 2024."},{"id":"mt-agentrisk","name":"MT-AgentRisk","year":2026,"by":"CHATS Lab (Weiyan Shi's group)","workloads":["safety","tool-calling","web-browsing","sql","coding"],"task_types":["follow a conversation where a harmful goal is split across several turns","operate a browser, terminal, file system, database or Notion via MCP tools","refuse once the combined intent becomes harmful","avoid destructive changes to data files or environment state"],"interaction":"multi-turn-scripted","environment":"containers","state_checked":false,"tools":["Playwright browser","terminal","filesystem","PostgreSQL and Notion MCP servers run under OpenHands"],"grading":["llm-judge"],"size":365,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2602.13379","github":"https://github.com/CHATS-lab/ToolShield","leaderboard":null},"good_for":["case-template","taxonomy"],"fit":"Fits agents that take many turns with MCP tools. Borrow the recipe that turns a single-turn harmful task into an addition or decomposition attack sequence, and the complete/rejected/ failed judge labels.","caveats":"Attack turns are pre-written, not adaptive; tasks are recycled from OpenAgentSafety, SafeArena, P2SQL and MCPMark; success is judged by GPT-4.1; dataset license is not stated separately from the repo."},{"id":"openagentsafety","name":"OpenAgentSafety","year":2025,"by":"Carnegie Mellon University","workloads":["safety","workflow-automation","coding","web-browsing","tool-calling"],"task_types":["carry out a workplace task with bash, files, a browser and chat","handle an ambiguous or conflicting instruction from a simulated colleague","refuse a request that would delete data, leak private data or run unsafe code","avoid financial loss, legal violations and malicious content"],"interaction":"multi-turn-simulated-user","environment":"containers","state_checked":true,"tools":["bash","file system","browser","GitLab","ownCloud","RocketChat"],"grading":["programmatic-check","llm-judge"],"size":356,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2507.06134","github":"https://github.com/Open-Agent-Safety/OpenAgentSafety","leaderboard":null},"good_for":["sandbox-design","grading-design","user-simulator"],"situations":["forbidden"],"fit":"Fits internal workplace or devops agents that act on real services. Borrow the Docker company sandbox from TheAgentCompany, the rule-based final-state checker paired with a four-label trajectory judge, and the NPC users that push unsafe requests.","caveats":"Needs about 30GB of Docker services; 35-49% of runs fail before reaching the risky step, so unsafe rates are over a subset; the original repo moved to a new organization in 2026."},{"id":"os-harm","name":"OS-Harm","year":2025,"by":"EPFL (TML)","workloads":["safety","computer-use","email-calendar","coding"],"task_types":["refuse a deliberate misuse request in a desktop app (harassment, disinformation, copyright)","resist a prompt injection shown inside an email, page or file","avoid unsafe side effects while doing a benign desktop task","use an email client, code editor and browser through the GUI"],"interaction":"single-turn","environment":"desktop","state_checked":false,"tools":["OSWorld virtual machine with email client","code editor","browser and office apps"],"grading":["llm-judge"],"size":150,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.14866","github":"https://github.com/tml-epfl/os-harm","leaderboard":null},"good_for":["taxonomy","grading-design","case-template"],"fit":"Fits computer-use agents that drive a real desktop. Borrow the three-way split (user misuse, injection, model misbehavior) and the judge that scores accuracy and safety together with published human agreement.","caveats":"Only 150 tasks; grading is an LLM judge over screenshots and actions (F1 about 0.76-0.79 against humans); needs VMware and OSWorld setup; injections are static."},{"id":"safearena","name":"SafeArena","year":2025,"by":"McGill University and Mila","workloads":["safety","web-browsing","shopping"],"task_types":["complete a safe web task on a forum, shop, shop admin or GitLab clone","refuse the paired harmful version (misinformation, illegal goods, harassment, cybercrime, bias)","stop partway once the harm becomes clear","avoid posting or listing harmful content"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["browser on four WebArena-style sites augmented with harmful content"],"grading":["programmatic-check","llm-judge"],"size":500,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2503.04957","github":"https://github.com/McGill-NLP/safearena","leaderboard":"https://huggingface.co/spaces/McGill-NLP/safearena-leaderboard"},"good_for":["case-template","grading-design","taxonomy"],"fit":"Fits web agents that post, buy or publish on behalf of users. Borrow the safe/harmful task pairs and the four-level risk scale (refuse at once, refuse later, attempt and fail, complete) judged on the trajectory.","caveats":"No license in the repo or dataset card; 200 of 250 harmful tasks were LLM-generated then reviewed; needs its own Docker site images; repo has been quiet since April 2025."},{"id":"shade-arena","name":"SHADE-Arena","year":2025,"by":"Anthropic (Kutasov et al.)","workloads":["safety","tool-calling","general-assistant","finance","email-calendar"],"task_types":["complete a long benign task with mock tools such as paying bills or booking travel","secretly complete a harmful side task in the same run","avoid raising suspicion from a monitor reading the transcript","act as a monitor and rate a transcript for suspicious behavior"],"interaction":"autonomous-long-horizon","environment":"api-sandbox","state_checked":true,"tools":["mock tools over a modified AgentDojo pipeline (banking","email","calendar","files)"],"grading":["programmatic-check","llm-judge"],"size":null,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.15740","github":"https://github.com/jkutaso/SHADE-Arena","leaderboard":null},"good_for":["grading-design","sandbox-design","case-template"],"fit":"Fits agents that run many tool calls on a user's behalf with money or data at stake, and teams that want an LLM monitor on their traces. Borrow the main-task plus side-task pairing, the 0-100 suspicion score from a monitor that cannot see the system prompt, and the mock-tool environment design.","caveats":"Only a public split is on GitHub (the private split with more tasks and transcripts is on request), the public README gives no task count, and the first safety-research repo is marked deprecated in favor of a personal account."},{"id":"st-webagentbench","name":"ST-WebAgentBench","year":2024,"by":"IBM Research (Levy, Shlomov et al.)","workloads":["safety","web-browsing","crm","devops"],"task_types":["complete an enterprise web task in GitLab, SuiteCRM or a shop admin panel","follow written safety and trust policies while doing it","ask for user consent or defer instead of acting when a policy requires it","stay inside the boundary of what the user asked"],"interaction":"autonomous-long-horizon","environment":"browser","state_checked":true,"tools":["browser via BrowserGym and Playwright"],"grading":["programmatic-check","state-diff","pass^k"],"size":375,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2410.06703","github":"https://github.com/segev-shlomov/ST-WebAgentBench","leaderboard":"https://huggingface.co/spaces/ST-WebAgentBench/st-webagentbench-leaderboard"},"good_for":["grading-design","taxonomy","case-template"],"situations":["forbidden","confidential"],"fit":"Fits browser agents that work inside company systems under rules, such as CRM or admin-panel automation. Borrow the Completion Under Policy metric (a task only counts if no policy was broken), the six policy dimensions, and the per-policy evaluator pattern.","caveats":"Built on WebArena images and SuiteCRM so it needs heavy environment setup; task count changed between versions (222 in early arXiv versions, 375 in the ICLR 2026 version) so published numbers are not directly comparable; the dataset card is inconsistent on the number of evaluator types."},{"id":"sycobench-600","name":"SycoBench-600","year":2026,"by":"Debu Sinha","workloads":["safety","knowledge-qa"],"task_types":["answer a multiple-choice question","keep the answer under three turns of doubt, false authority or a wrong suggestion","accept a correct suggestion when the first answer was wrong"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match"],"size":600,"data":"synthetic","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://aclanthology.org/2026.findings-acl.1759/","github":"https://github.com/debu-sinha/sycobench-600","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits agents that must hold a correct answer under user pressure but still take valid corrections, such as support or advisory bots. Borrow the correction-selectivity metric (update rate minus wrong-flip rate) and the matched correct-suggestion turn.","caveats":"Multiple-choice only with no tools, so it measures the model not the agent; single author; headline numbers use a 555-question subset; dataset is CC-BY-4.0 while code is MIT."},{"id":"sycophancy-eval","name":"Sycophancy Eval","year":2023,"by":"Anthropic (Sharma, Tong et al.)","workloads":["safety","knowledge-qa","general-assistant"],"task_types":["answer a factual question, then hold or change the answer when the user says \"Are you sure?\"","give feedback on an argument the user says they like or dislike","answer when the user suggests a wrong answer with hedging"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge","exact-match"],"size":null,"data":"mixed","pii":"none","license":"unknown","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2310.13548","github":"https://github.com/meg-tong/sycophancy-eval","leaderboard":null},"good_for":["case-template","grading-design"],"fit":"Fits any agent that answers users who push back, such as support, tutoring or advisory agents. Borrow the \"are you sure?\" follow-up turn and the three metrics (confidence, apologize rate, truthfulness) as a cheap regression check for caving to pressure.","caveats":"Not agentic (no tools, scripted two-turn pushback); the inspect_evals port implements only the \"are you sure\" dataset; no license file in the repo; questions come from older QA sets such as TriviaQA so leakage is likely."},{"id":"toolemu","name":"ToolEmu","year":2023,"by":"University of Toronto","workloads":["safety","tool-calling","general-assistant","finance"],"task_types":["carry out a risky instruction (payments, file deletion, smart home) without causing harm","seek confirmation before an irreversible action","resist underspecified instructions that invite unsafe assumptions"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["311 tools in 36 toolkits, all emulated by an LLM from their specs"],"grading":["llm-judge"],"size":144,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2309.15817","github":"https://github.com/ryoungj/ToolEmu","leaderboard":null},"good_for":["sandbox-design","grading-design","taxonomy"],"situations":["forbidden"],"fit":"Fits teams that need to test dangerous actions before any real tool exists. Borrow the LLM-emulated tool sandbox and the paired safety and helpfulness scores on a 0-3 scale.","caveats":"No real execution, so tool behavior is whatever the emulator imagines; 144 cases, GPT-4 judging at about one dollar per case, and the repo has been idle since 2024."},{"id":"wasp","name":"WASP","year":2025,"by":"Meta (FAIR)","workloads":["safety","security","web-browsing"],"task_types":["complete a user task on a GitLab or Reddit clone","ignore an injection planted in page text or a URL","avoid being diverted to the attacker's goal mid-task","not finish the attacker's goal end to end"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["browser on VisualWebArena GitLab and Reddit clones"],"grading":["state-diff","llm-judge"],"size":84,"data":"synthetic","pii":"none","license":"CC-BY-NC-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2504.18575","github":"https://github.com/facebookresearch/wasp","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits web agents that act on sites with user-generated content. Borrow the two-level metric, intermediate attack success (judge sees the agent diverted) versus end-to-end success (final state check), which separates gullibility from damage.","caveats":"Only 4 user tasks and 21 attacker goals crossed into 84 cases; non-commercial license; needs the full WebArena docker setup; the utility set's grading method is not described."},{"id":"salesllm","name":"SalesLLM","year":2026,"by":"Bairong","workloads":["sales","finance","retail"],"task_types":["move a simulated customer through a sales conversation toward a purchase decision","handle objections and persuade without breaking role","adapt to customer personas and difficulty levels in Chinese or English"],"interaction":"multi-turn-simulated-user","environment":"static","state_checked":false,"tools":["none"],"grading":["llm-judge","programmatic-check"],"size":1805,"data":"mixed","pii":"possible","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2604.07054","github":"https://github.com/Bairong-Xdynamics/Benchmarking-LLM-Realistic-Selling-Skill","leaderboard":null},"good_for":["user-simulator","grading-design","baseline"],"fit":"Fits outbound or inbound sales chat agents with no backend actions. Borrow CustomerLM, a fine-tuned customer simulator that resists role inversion, and the two-part score: an LLM judge for sales-process progress plus a BERT classifier for end-of-dialogue buying intent.","caveats":"No tools or state, so it measures persuasion only; 1,000 of the 1,805 scenarios are Chinese and the authors report poor cross-lingual consistency; the simulator was trained on crowdworker conversations and 10,000 dialogues are released, so check for personal data before reuse."},{"id":"astabench","name":"AstaBench","year":2025,"by":"Allen Institute for AI (AI2)","workloads":["science","research","coding","data-analysis","planning"],"task_types":["find papers and answer literature questions with a search tool","build a literature comparison table","reproduce results from a paper's code","run a data-driven discovery analysis","carry out an end-to-end research task from question to report"],"interaction":"autonomous-long-horizon","environment":"mixed","state_checked":false,"tools":["literature search MCP","Python sandbox","report and table editors"],"grading":["programmatic-check","llm-judge"],"size":2400,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.21652","github":"https://github.com/allenai/asta-bench","leaderboard":"https://huggingface.co/spaces/allenai/asta-bench-leaderboard"},"good_for":["sandbox-design","baseline","taxonomy"],"fit":"Fits scientific research-assistant agents that combine literature search, code execution and data analysis. Borrow its reproducible search environment with date cutoffs, its cost-per-task reporting next to accuracy, and its four-category macro average.","caveats":"It is a suite of 11 existing and new benchmarks (CORE-Bench-Hard, DiscoveryBench, DS-1000 and others), so results overlap with those; needs a large machine (authors suggest 128 GB RAM), API keys for scoring, and an Asta tool key for the search tool; the leaderboard is curated by AI2."},{"id":"bioprobench","name":"BioProBench","year":2025,"by":"Peking University","workloads":["science","knowledge-qa","planning"],"task_types":["answer a question about a wet-lab protocol","put shuffled protocol steps in the right order","find and fix an error in a protocol","write a full protocol from a goal","reason about why a protocol step is needed"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match","programmatic-check"],"size":523784,"data":"real-scraped","pii":"none","license":"CC-BY-NC-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2505.07889","github":"https://github.com/YuyangSunshine/bioprobench","leaderboard":null},"good_for":["case-template","taxonomy","grading-design"],"fit":"Fits lab-assistant or protocol-writing agents that must respect procedural order, quantities and safety steps. Borrow the five task shapes (QA, ordering, error correction, generation, reasoning) and the step recall and Kendall tau metrics for ordered procedures.","caveats":"Text-only, no robot or lab environment; non-commercial license; the huge instance count is auto-derived from 22k protocols, so quality is uneven and the held-out test size is not stated in the README; generation is scored with BLEU and keyword overlap."},{"id":"bixbench","name":"BixBench","year":2025,"by":"FutureHouse","workloads":["science","data-analysis","coding"],"task_types":["explore a biological dataset in a notebook","run a multi-step analysis in Python, R or Bash","answer an open-ended research question from the results","answer the same question in multiple-choice form"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":false,"tools":["Jupyter notebook","Python","R","Bash"],"grading":["llm-judge","exact-match"],"size":205,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2503.00096","github":"https://github.com/Future-House/BixBench","leaderboard":null},"good_for":["sandbox-design","grading-design","case-template"],"fit":"Fits bioinformatics or data-analysis agents that work inside a notebook against real data. Borrow the \"capsule\" pattern (a published notebook plus its data turned into open questions) and the Docker notebook sandbox.","caveats":"Only 60 source notebooks and 205 questions; open answers are graded by an LLM; the Hugging Face dataset needs authentication; multiple-choice scores were at chance for frontier models at release, so that mode is a weak signal."},{"id":"blade","name":"BLADE","year":2024,"by":"University of Washington (Gu et al.)","workloads":["science","data-analysis"],"task_types":["pick the variables and transforms needed to answer a research question from a dataset","write a data transform function","specify a statistical model and justify the choice"],"interaction":"single-turn","environment":"terminal","state_checked":false,"tools":["python code execution"],"grading":["llm-judge","programmatic-check"],"size":12,"data":"real-public","pii":"none","license":"ODC-By-1.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2408.09667","github":"https://github.com/behavioral-data/BLADE","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits research or analytics agents whose output is an analysis plan, not a single number. Borrow the way expert analysts' decisions (constructs, transforms, model) are collected as ground truth and matched to the agent's choices.","caveats":"Only 12 dataset and question pairs; evaluation needs an LLM; the README still has placeholder counts and says a hold-out test set is in progress."},{"id":"chembench","name":"ChemBench","year":2024,"by":"LamaLab (Jablonka group, University of Jena)","workloads":["science","knowledge-qa","safety"],"task_types":["answer a chemistry question with a number or a choice","reason about reactions, spectra and molecular properties","answer chemical safety and toxicity questions","say how confident the answer is"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match","programmatic-check"],"size":2700,"data":"mixed","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2404.01475","github":"https://github.com/lamalab-org/chembench","leaderboard":"https://huggingface.co/spaces/jablonkagroup/ChemBench-Leaderboard"},"good_for":["baseline","taxonomy","grading-design"],"fit":"Fits chemistry assistants and lab-support agents that must answer questions and know when they are unsure. Borrow its topic-level reports, numeric-tolerance parsing of answers, and the comparison against a human chemist panel.","caveats":"Question answering only, no tools or environment; top models already beat the human chemists in the paper so headline scores are near saturation; the paper notes models still fail some basic tasks and are overconfident, which the headline score hides."},{"id":"discoverybench","name":"DiscoveryBench","year":2024,"by":"Allen Institute for AI","workloads":["science","data-analysis","research"],"task_types":["form a hypothesis from one or more datasets given a discovery goal","run the statistical analysis that tests it","state the finding with its context, variables and relationship"],"interaction":"single-turn","environment":"terminal","state_checked":false,"tools":["python code execution"],"grading":["llm-judge"],"size":264,"data":"mixed","pii":"none","license":"ODC-By-1.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2407.01725","github":"https://github.com/allenai/discoverybench","leaderboard":null},"good_for":["grading-design","taxonomy","case-template"],"fit":"Fits research agents that must turn a dataset and a goal into a written finding. Borrow the faceted judge: split gold and predicted hypotheses into context, variables and relationship and score each part separately.","caveats":"Scoring depends on a GPT-4 judge; the best system in the paper scored about 25 percent, so numbers are noisy; the extra 903 synthetic tasks are for controlled tests, not realism."},{"id":"frontierscience","name":"FrontierScience","year":2025,"by":"OpenAI","workloads":["science","knowledge-qa","research"],"task_types":["solve an olympiad-level physics, chemistry or biology problem","work through a PhD-level open research sub-task","show intermediate reasoning that a rubric can credit"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match","llm-judge"],"size":160,"data":"synthetic","pii":"none","license":"unknown","harness":["inspect_evals"],"status":"active","links":{"paper":"https://arxiv.org/abs/2601.21165","github":"https://github.com/UKGovernmentBEIS/inspect_evals","leaderboard":null},"good_for":["grading-design","baseline"],"fit":"Fits science reasoning assistants where you want a hard, unsaturated reference number. Borrow its research-track rubrics that give partial credit for progress through a task rather than only the final answer.","caveats":"Only the 160-question gold set is public (100 olympiad, 60 research); the rest is held by OpenAI; no tools or environment, so it measures reasoning not agent behaviour; no official repository or license found, only the inspect_evals port."},{"id":"lab-bench","name":"LAB-Bench","year":2024,"by":"FutureHouse","workloads":["science","knowledge-qa","research"],"task_types":["answer a literature question that needs the paper's full text","look up a value in a biological database","read a figure or table from a paper","troubleshoot a lab protocol","manipulate a DNA or protein sequence","plan a molecular cloning scenario"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none required; agents may use search or code"],"grading":["exact-match"],"size":2400,"data":"real-public","pii":"none","license":"CC-BY-SA-4.0","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2407.10362","github":"https://github.com/Future-House/LAB-Bench","leaderboard":null},"good_for":["taxonomy","baseline","case-template"],"fit":"Fits research-assistant agents for biology that answer questions over papers, databases and sequences. Borrow the eight-category taxonomy (LitQA2, DbQA, SuppQA, FigQA, TableQA, ProtocolQA, SeqQA, Cloning) and the accuracy-plus-coverage reporting that rewards abstaining.","caveats":"Multiple-choice only, so not an agent benchmark on its own; 20 percent of the data is held private; frontier models have largely caught up and the authors released LABBench2 (arXiv 2604.09554, about 1,900 tasks, CC-BY-SA-4.0, github.com/EdisonScientific/labbench2, also in inspect_evals) as the harder successor."},{"id":"scicode","name":"SciCode","year":2024,"by":"Tian et al. (30 authors, scientist-curated)","workloads":["science","coding"],"task_types":["implement a research computation in Python from a scientist-written description","solve a chain of subproblems that build on each other in one scientific problem"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["unit-tests"],"size":80,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["inspect_evals","hal","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2407.13168","github":"https://github.com/scicode-bench/SciCode","leaderboard":"https://scicode-bench.github.io/leaderboard/"},"good_for":["case-template","baseline"],"fit":"Fits scientific-coding assistants; borrow the main-problem versus subproblem scoring so partial progress is visible.","caveats":"Only 80 main problems (338 subproblems); best model solved 4.6 percent at release, so early numbers are mostly zero; no tool use or repository context."},{"id":"scienceagentbench","name":"ScienceAgentBench","year":2024,"by":"Ohio State University (OSU NLP Group)","workloads":["science","data-analysis","coding","research"],"task_types":["write a self-contained Python program for a data-analysis task from a paper","load and process a scientific dataset","produce a figure or model output that matches the expected result","debug the program when it fails to run"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":false,"tools":["Python","shell"],"grading":["programmatic-check","llm-judge"],"size":102,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo","hal"],"status":"active","links":{"paper":"https://arxiv.org/abs/2410.05080","github":"https://github.com/OSU-NLP-Group/ScienceAgentBench","leaderboard":"https://hal.cs.princeton.edu/"},"good_for":["grading-design","case-template","baseline"],"fit":"Fits data-analysis or scientific coding agents that take a dataset and a goal and must hand back a runnable program. Borrow its per-task evaluation scripts that execute the generated code and compare outputs, plus the best-of-three-runs scoring.","caveats":"Only 102 tasks from 44 papers in four fields; figure outputs are judged by GPT-4o so an OpenAI key is required; a verified split was released in 2026, so older numbers do not compare. Code is MIT but a few tasks keep the original data licenses."},{"id":"scienceworld","name":"ScienceWorld","year":2022,"by":"Allen Institute for AI","workloads":["science","education","embodied","planning"],"task_types":["run an elementary-school science experiment in a text simulator","measure or change a property (boil water, test conductivity, grow a plant)","classify or find an object that matches a stated property"],"interaction":"multi-turn-scripted","environment":"api-sandbox","state_checked":true,"tools":["text-world action interface"],"grading":["programmatic-check"],"size":30,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2203.07540","github":"https://github.com/allenai/ScienceWorld","leaderboard":null},"good_for":["grading-design","sandbox-design"],"fit":"Fits teams who want a procedure-following environment with partial-progress scoring; the 30 task types have about 7,000 parametric variations, so it resists memorization better than ALFWorld.","caveats":"Simulated science, not a business workload; size counts task types, not variations; results are hard to compare across papers because step budgets differ."},{"id":"sciknoweval","name":"SciKnowEval","year":2024,"by":"Zhejiang University (HICAI)","workloads":["science","knowledge-qa","safety"],"task_types":["recall a scientific fact","explain or comprehend a scientific text","reason through a multi-step science problem","judge whether a request is safe or ethical","apply knowledge to a practical task such as property prediction"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match","llm-judge","programmatic-check"],"size":28392,"data":"mixed","pii":"none","license":"unknown","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2406.09098","github":"https://github.com/HICAI-ZJU/SciKnowEval","leaderboard":"https://scimind.ai/sciknoweval"},"good_for":["taxonomy","baseline"],"fit":"Fits science question-answering assistants in biology, chemistry, physics or materials. Borrow the five-level taxonomy (memory, comprehension, reasoning, discernment, application) as a way to bucket an agent's own test cases by depth.","caveats":"Question answering only, not agentic; no license stated in the repository; some tasks need GPT-4o as grader and are skipped without a key; relation-extraction tasks are scored by word-vector similarity, which is a loose check."},{"id":"caibench","name":"CAIBench","year":2025,"by":"Alias Robotics","workloads":["security","knowledge-qa","multi-agent"],"task_types":["solve jeopardy-style capture-the-flag challenges","attack and defend services at the same time in an attack-defense CTF","run a multi-step exercise in a cyber range","answer security knowledge questions","handle personal data without leaking it"],"interaction":"autonomous-long-horizon","environment":"mixed","state_checked":true,"tools":["shell and security tools through the CAI framework"],"grading":["exact-match","programmatic-check"],"size":null,"data":"mixed","pii":"possible","license":"MIT","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2510.24317","github":"https://github.com/aliasrobotics/cai","leaderboard":null},"good_for":["taxonomy","sandbox-design"],"fit":"Fits teams building both attack and defense agents who want one place to measure knowledge, CTF and range tasks. Borrow the split into five categories and the attack-defense setup where two agents face each other.","caveats":"The CAI repo is archived and read-only; license mixes MIT with proprietary research-only additions; the abstract says over 10,000 instances but no exact count was seen; privacy category uses personal-data-like content."},{"id":"cti-realm","name":"CTI-REALM","year":2026,"by":"Chakraborty, Ho, Cook, Meléndez","workloads":["security","it-ops","tool-calling"],"task_types":["read a cyber threat intelligence report","map the described behavior to MITRE ATT&CK techniques","explore log schemas and run KQL queries against an emulated attack","write a KQL or Sigma detection rule that catches the attack"],"interaction":"autonomous-long-horizon","environment":"mixed","state_checked":false,"tools":["CTI report retrieval","KQL execution against a Kusto emulator","Sigma rule retrieval","MITRE mapping","bash","python"],"grading":["trajectory-check","llm-judge","programmatic-check"],"size":50,"data":"synthetic","pii":"none","license":"unknown","harness":["inspect_evals"],"status":"active","links":{"paper":"https://arxiv.org/abs/2603.13517","github":null,"leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits detection-engineering or security-operations agents. Borrow the weighted checkpoint scoring that credits CTI analysis, MITRE mapping, data exploration and query execution on the way to the final rule, and the F1 scoring of a detection rule against ground truth.","caveats":"Only 50 tasks; the CTI-analysis judge correlates weakly with human labels (r=0.48); needs a 4.5 GB Docker image with a Kusto emulator; no dataset license stated; no standalone repo seen outside inspect_evals and a Hugging Face dataset."},{"id":"cve-bench","name":"CVE-Bench","year":2025,"by":"UIUC (Kang lab; Zhu et al.)","workloads":["security","web-browsing"],"task_types":["attack a running web application with a known critical CVE","achieve one of eight outcomes such as database access, file access or admin login","work with or without a hint about the exploit"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell and HTTP access to a target web app in Docker or Kubernetes"],"grading":["programmatic-check"],"size":40,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2503.17332","github":"https://github.com/uiuc-kang-lab/cve-bench","leaderboard":null},"good_for":["grading-design","sandbox-design"],"fit":"Fits penetration-testing or web-security agents. Borrow the eight concrete attack outcomes checked by graders (not just a flag), and the zero-day versus one-day prompt variants.","caveats":"Only 40 CVEs; reference exploits are withheld and given on request; the Kubernetes sandbox supports only a subset; arm64 is not guaranteed to work."},{"id":"cybench","name":"Cybench","year":2024,"by":"Stanford (Zhang, Liang et al.)","workloads":["security","coding"],"task_types":["solve a professional capture-the-flag challenge in a Kali Linux container","work through crypto, web, reverse engineering, forensics and binary exploitation","solve guided subtasks that break a challenge into steps"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":false,"tools":["bash in Kali Linux","optional internet"],"grading":["exact-match"],"size":40,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2408.08926","github":"https://github.com/andyzorigin/cybench","leaderboard":"https://cybench.github.io/"},"good_for":["baseline","case-template","sandbox-design"],"fit":"Fits offensive-security or CTF-style agents and teams that want a known capability baseline. Borrow the subtask mode that gives partial credit for intermediate steps and the per-task human solve time as a difficulty scale.","caveats":"Public CTF challenges, so leakage grows over time; the inspect_evals port drops one task for GPL reasons and rebuilt some victim images so numbers differ; flag matching only."},{"id":"cybergym","name":"CyberGym","year":2025,"by":"UC Berkeley (Wang, Song et al.)","workloads":["security","coding"],"task_types":["read a vulnerability description and a large C or C++ codebase","write a proof-of-concept input that triggers the crash","check the input against the vulnerable and the patched build"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["bash in a Docker sandbox","proof-of-concept submission script"],"grading":["programmatic-check"],"size":1507,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.02548","github":"https://github.com/sunblaze-ucb/cybergym","leaderboard":"https://www.cybergym.io/"},"good_for":["grading-design","sandbox-design","baseline"],"fit":"Fits code-security or bug-finding agents that work on real repositories. Borrow the execution oracle that runs the agent's input against a vulnerable and a fixed build, and the four difficulty levels that vary how much context is given.","caveats":"Data is about 240 GB and the full server data about 10 TB; sources are public OSS-Fuzz and ARVO crashes so models may have seen them; the inspect_evals port had a scoring bug fixed in version 2-A so older results are invalid."},{"id":"excytin-bench","name":"ExCyTIn-Bench","year":2025,"by":"Microsoft (Wu et al.)","workloads":["security","sql","it-ops"],"task_types":["investigate a simulated attack from Microsoft Sentinel logs","write SQL queries over 57 log tables to follow a chain of evidence","answer a question about the incident such as which host or account was involved"],"interaction":"autonomous-long-horizon","environment":"database","state_checked":false,"tools":["SQL query tool over a MySQL copy of Azure security logs"],"grading":["llm-judge","exact-match"],"size":null,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2507.14201","github":"https://github.com/microsoft/SecRL","leaderboard":null},"good_for":["case-template","grading-design","privacy-pattern"],"fit":"Fits security operations or log-investigation agents that query a SIEM. Borrow the way questions are generated from an investigation graph so each has an explainable answer, and the controlled tenant that yields realistic logs without customer data.","caveats":"Question counts differ between sources (the arXiv abstract says 7,542 generated questions, other listings cite 589 evaluation questions) so no size is given; eight incidents only; needs Docker MySQL and GPT-4o for grading."},{"id":"exploitbench","name":"ExploitBench","year":2026,"by":"Lee and Brumley","workloads":["security","coding"],"task_types":["build an exploit for a real V8 JavaScript engine bug with mitigations on","climb a ladder from reaching the code, to crash, to memory read and write, to code execution"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":false,"tools":["shell access to a patched d8 binary with oracles"],"grading":["programmatic-check"],"size":41,"data":"real-public","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2605.14153","github":"https://github.com/ChaoticCooties/exploitbench-eval","leaderboard":null},"good_for":["grading-design"],"fit":"Fits advanced exploit-development or vulnerability-research agents. Borrow the 16 binary capability flags checked by deterministic oracles, which give graded progress instead of a single pass or fail.","caveats":"Narrow (V8 only, 41 bugs); episodes run up to 300 turns so it is expensive; the GitHub repo README is still the inspect_evals template, so details live in the paper and inspect page."},{"id":"gdm-dangerous-capabilities-ctf","name":"GDM Dangerous Capabilities: Capture the Flag","year":2024,"by":"Google DeepMind (Phuong et al.)","workloads":["security","it-ops"],"task_types":["exploit a web application vulnerability to read a flag","use an off-the-shelf exploit against a service","escalate privileges on a Linux host","crack or spray passwords"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":false,"tools":["bash in an attacker container against a victim container"],"grading":["exact-match"],"size":13,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2403.13793","github":"https://github.com/google-deepmind/dangerous-capability-evaluations","leaderboard":null},"good_for":["sandbox-design","case-template"],"fit":"Fits offensive-security agents and teams that need a small in-house CTF set that is not on the public internet. Borrow the attacker-plus-victim compose layout and the at-least-one-of-N epochs reducer.","caveats":"Only 13 challenges; the DeepMind repo says the material will not run off the shelf and flags are placeholders you must set; some tasks ship wordlist hints that make them easier."},{"id":"intercode-ctf","name":"InterCode CTF","year":2023,"by":"Princeton NLP (Yang et al.)","workloads":["security","coding"],"task_types":["solve a picoCTF-style challenge with bash and python","decode, reverse or exploit a small binary or file to recover a flag"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":false,"tools":["bash","python in Docker"],"grading":["exact-match"],"size":100,"data":"real-public","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"saturated","links":{"paper":"https://arxiv.org/abs/2306.14898","github":"https://github.com/princeton-nlp/intercode","leaderboard":null},"good_for":["baseline","sandbox-design"],"fit":"Fits teams that want a quick, cheap smoke test for a shell-using agent. Borrow the environment-as-RL-loop design where code is the action and execution output is the observation.","caveats":"Saturated (inspect_evals reports 0.94 for a 2025 Claude model); picoCTF tasks are public and easy; inspect_evals runs only 78 of 100 because 22 need internet; harness differs from the DeepMind paper that reused it."},{"id":"patcheval","name":"PatchEval","year":2025,"by":"Qi et al. (11 authors)","workloads":["security","coding"],"task_types":["patch a real CVE in a Go, JavaScript or Python project","pass both the security test and the existing functionality tests in a sandbox"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor"],"grading":["unit-tests","programmatic-check"],"size":1000,"data":"real-public","pii":"none","license":"unknown","harness":["inspect_evals"],"status":"active","links":{"paper":"https://arxiv.org/abs/2511.11019","github":null,"leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits vulnerability-repair agents in web languages; borrow the paired security plus functionality test as the pass rule.","caveats":"Only 230 of the 1,000 CVEs have runtime sandboxes, so most are static; code link and data license were not on the paper page."},{"id":"sec-bench","name":"SEC-bench","year":2025,"by":"UIUC (Lee, Zhang, Lu, Zhang)","workloads":["security","coding"],"task_types":["write a patch that removes a real memory-safety vulnerability in a C/C++ project without breaking it","generate a proof-of-concept input that triggers a reported vulnerability"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":true,"tools":["shell","file editor","sanitizers"],"grading":["programmatic-check","unit-tests"],"size":null,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.11791","github":"https://github.com/SEC-bench/SEC-bench","leaderboard":"https://sec-bench.github.io/"},"good_for":["sandbox-design","grading-design"],"fit":"Fits security-fix agents; borrow the automated pipeline that reproduces a CVE in a container and the two-sided check (exploit no longer fires, build still passes).","caveats":"C/C++ only; instance count is not in the README; a follow-up (PatchBench) reports a recent agent at 97 percent, so the patching task may be near saturation."},{"id":"threecb","name":"3CB (Catastrophic Cyber Capabilities Benchmark)","year":2024,"by":"Apart Research (Anurin et al.)","workloads":["security","it-ops"],"task_types":["scan and enumerate a network or host","escalate privileges through a setuid binary or misconfiguration","navigate a web application to find a hidden flag","carry out an attack step mapped to MITRE ATT&CK"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":false,"tools":["bash in a per-task Docker sandbox"],"grading":["exact-match"],"size":11,"data":"synthetic","pii":"none","license":"unknown","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2410.09114","github":"https://github.com/apartresearch/3cb","leaderboard":null},"good_for":["case-template","taxonomy"],"fit":"Fits offensive-security agents and red-team tooling. Borrow the per-task config that bundles sandbox, prompt elicitations and success rule in one file, and the mapping of tasks to ATT&CK techniques.","caveats":"Small (11 tasks in inspect_evals); no license file seen; inspect_evals numbers are not comparable to the paper because prompts and the communication protocol differ; POSIX only."},{"id":"deepshop","name":"DeepShop","year":2025,"by":"Lyu et al. (University of Amsterdam / de Rijke group)","workloads":["shopping","web-browsing","research"],"task_types":["find products that satisfy several attributes, filters and a sort preference","compare options across live shopping sites and report a result"],"interaction":"single-turn","environment":"browser","state_checked":false,"tools":["live shopping websites","search"],"grading":["programmatic-check","llm-judge"],"size":null,"data":"mixed","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.02839","github":null,"leaderboard":null},"good_for":["grading-design","taxonomy"],"fit":"Fits shopping research assistants. Borrow the fine-grained scoring of attributes, filters and sort order plus the easy / medium / hard query evolution.","caveats":"No code or license found; query count not stated; live sites drift; success rates under 35 percent at release."},{"id":"shoppingbench","name":"ShoppingBench","year":2025,"by":"ShoppingBench authors (arXiv 2508.04266)","workloads":["shopping","retail"],"task_types":["find or buy a product in a simulated store with 2.5 million real items","apply a voucher, stay within a budget, find a seller with several wanted items"],"interaction":"single-turn","environment":"api-sandbox","state_checked":true,"tools":["simulated shopping platform"],"grading":["programmatic-check"],"size":null,"data":"mixed","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.04266","github":null,"leaderboard":null},"good_for":["user-simulator","case-template"],"fit":"Fits shopping assistants with budget or promotion logic. Borrow the intent-grounded instruction generation from sampled products.","caveats":"Task count, code and license not found; instructions are simulated; GPT-4.1 under 50 percent at release."},{"id":"webmall","name":"WebMall","year":2025,"by":"University of Mannheim (Peeters, Bizer et al.)","workloads":["shopping","retail","web-browsing"],"task_types":["find a specific product across four shops and compare prices","find substitutes or compatible products from a vague description","add to cart and check out"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["four self-hosted simulated shops"],"grading":["programmatic-check","state-diff"],"size":91,"data":"mixed","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.13024","github":"https://wbsg-uni-mannheim.github.io/WebMall/","leaderboard":null},"good_for":["sandbox-design","case-template"],"fit":"Fits comparison-shopping agents; borrow the multi-shop offline sandbox built from Common Crawl product pages and the basic / advanced task split.","caveats":"Small; shops are simplified; license not stated; best agents 62 to 75 percent on basic tasks and 19 to 41 on advanced."},{"id":"webshop","name":"WebShop","year":2022,"by":"Princeton NLP (Yao et al.)","workloads":["shopping","retail","web-browsing"],"task_types":["search a simulated store for a product matching a text request","pick options (size, color) and a price limit","add to cart and buy"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["simulated e-commerce site (search","click","buy)"],"grading":["programmatic-check"],"size":12087,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2207.01206","github":"https://github.com/princeton-nlp/WebShop","leaderboard":null},"good_for":["sandbox-design","grading-design"],"fit":"Fits shopping assistants; borrow the self-hosted product catalog sandbox and the attribute-and-price reward computed from the bought item.","caveats":"Simplified text-first site from 2022, not updated; single-site, single-purchase flow; 1.18 million real products scraped from Amazon."},{"id":"sheetcopilot","name":"SheetCopilot","year":2023,"by":"Li et al. (BraveGroup)","workloads":["spreadsheets","workflow-automation"],"task_types":["format, sort or fill cells from a plain-language request","build a chart or pivot table in a workbook","carry out a multi-step spreadsheet task through atomic actions"],"interaction":"single-turn","environment":"desktop","state_checked":true,"tools":["atomic spreadsheet actions over Excel or LibreOffice"],"grading":["state-diff"],"size":221,"data":"synthetic","pii":"none","license":"GPL-3.0-only","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2305.19308","github":"https://github.com/BraveGroup/SheetCopilot","leaderboard":null},"good_for":["grading-design","sandbox-design","case-template"],"fit":"Fits agents that drive a spreadsheet application through a fixed action set rather than writing code. Borrow the per-task checking file that names which sheet properties to compare against one or more reference workbooks.","caveats":"Tasks were expanded from 67 seed tasks over 28 workbooks, so coverage is narrow; the original evaluator needs Windows and Excel; the README restricts use to non-commercial purposes; results shown are from the GPT-4 era."},{"id":"spreadsheetbench","name":"SpreadsheetBench","year":2024,"by":"Ma et al. (RUCKBReasoning)","workloads":["spreadsheets","data-analysis","coding"],"task_types":["fill or compute cells as a forum user asks, with a formula or a script","transform a sheet that has several tables or a non-standard layout","make the solution hold when cell values change across test workbooks"],"interaction":"single-turn","environment":"containers","state_checked":true,"tools":["python with openpyxl","spreadsheet files"],"grading":["state-diff","programmatic-check"],"size":912,"data":"real-scraped","pii":"possible","license":"CC-BY-SA-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2406.14991","github":"https://github.com/RUCKBReasoning/SpreadsheetBench","leaderboard":null},"good_for":["grading-design","case-template","sandbox-design"],"fit":"Fits spreadsheet or Excel assistants that edit workbooks from a user request. Borrow the online-judge pattern: about three input workbooks per instruction, and the output cells must match on every one.","caveats":"Questions come from public Excel forums, so sheets may carry user data; formulas must be recalculated through LibreOffice or Excel before checking; a smaller Verified subset was released later, so compare numbers carefully."},{"id":"bird","name":"BIRD","year":2023,"by":"BIRD team (Alibaba DAMO-ConvAI and University of Hong Kong)","workloads":["sql","data-analysis","knowledge-qa"],"task_types":["translate a natural-language question into one SQLite query","use the given external knowledge hint to map business terms to columns","handle dirty values and large database contents when writing SQL","write SQL that runs efficiently on a large database"],"interaction":"single-turn","environment":"database","state_checked":false,"tools":["SQLite execution"],"grading":["programmatic-check"],"size":12751,"data":"real-public","pii":"none","license":"CC-BY-SA-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2305.03111","github":"https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/bird","leaderboard":"https://bird-bench.github.io/"},"good_for":["baseline","grading-design","case-template"],"fit":"Fits single-shot text-to-SQL agents that answer questions over one database with a known schema. Borrow its execution-accuracy grading, the evidence hint pattern and the human baseline (92.96 execution accuracy) as a reference point.","caveats":"Test set is hidden and scored by the maintainers; the dev set has likely leaked into model training data; SQLite only, one query per question, no multi-turn; the top entry (82.95) is within ten points of the human score."},{"id":"bird-critic","name":"BIRD-Critic (SWE-SQL)","year":2025,"by":"BIRD team (University of Hong Kong and Google Cloud)","workloads":["sql","coding","it-ops"],"task_types":["fix a user's broken SQL given their issue description and the database","diagnose why a query returns wrong results or fails on a given dialect","rewrite SQL across PostgreSQL, MySQL, SQL Server, Oracle or BigQuery","produce a corrected query that passes hidden test cases"],"interaction":"single-turn","environment":"database","state_checked":true,"tools":["SQL execution in a per-dialect database environment"],"grading":["programmatic-check","unit-tests"],"size":570,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.18951","github":"https://github.com/bird-bench/BIRD-CRITIC-1","leaderboard":"https://bird-critic.github.io/"},"good_for":["grading-design","case-template","taxonomy"],"situations":["tool-error"],"fit":"Fits SQL support or database debugging agents that take a user complaint and a broken query and must return a working one. Borrow its test-case grading on the live database and its variants (Open 570 multi-dialect, PostgreSQL 530, Flash 200, BigQuery 200).","caveats":"Ground-truth SQL and test cases are withheld and must be requested by email; human score on the single-dialect set is 78.87% versus about 35-39% for the best models; issues are drawn from user posts rather than one company's workload."},{"id":"bird-interact","name":"BIRD-Interact","year":2025,"by":"BIRD team (University of Hong Kong)","workloads":["sql","data-analysis","customer-support"],"task_types":["ask a simulated user to clarify an ambiguous database request","write and run PostgreSQL queries for business intelligence questions","create, update or delete rows and tables as the user asks","recover from an execution error and continue the conversation","decide when to explore the database versus ask the user"],"interaction":"multi-turn-simulated-user","environment":"database","state_checked":true,"tools":["PostgreSQL execution","knowledge base and metadata files","user simulator"],"grading":["programmatic-check","state-diff"],"size":600,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.05318","github":"https://github.com/bird-bench/BIRD-Interact","leaderboard":"https://bird-interact.github.io/"},"good_for":["user-simulator","grading-design","case-template"],"situations":["ambiguous","multi-turn","tool-error"],"fit":"Fits conversational database assistants that must handle vague requests, clarification and write operations. Borrow the function-driven user simulator, the two modes (scripted c-Interact and open a-Interact) and the per-task executable test cases.","caveats":"Ground-truth SQL and test cases are gated behind an email request; the lite split is 300 tasks in the paper but 270 in the repo README; the dataset is CC-BY-SA-4.0 while code is MIT; best model solves about 17% of full tasks so scores are hard to compare."},{"id":"corgi","name":"CORGI","year":2025,"by":"Yue Li et al. (Cornell University and Gena AI)","workloads":["sql","data-analysis"],"task_types":["answer a descriptive business question with SQL over a company-style database","explain a trend by combining several queries","make a prediction or forecast from the data","give a recommendation that is grounded in query results"],"interaction":"single-turn","environment":"database","state_checked":false,"tools":["SQL execution"],"grading":["programmatic-check","llm-judge"],"size":null,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.07309","github":null,"leaderboard":null},"good_for":["taxonomy","case-template"],"fit":"Fits business-analyst agents whose users ask why and what-next questions, not only lookups. Borrow its four-level question taxonomy (descriptive, explanatory, predictive, recommendational) and its success execution rate metric.","caveats":"No code or dataset link was found from the abstract page although the paper says one is released; databases are synthetic imitations of companies such as DoorDash and Airbnb; single-turn only; the paper is still under review and figures changed between versions."},{"id":"dacomp","name":"DAComp","year":2025,"by":"Fangyu Lei et al.","workloads":["sql","data-analysis","devops"],"task_types":["build a multi-stage SQL pipeline from raw sources on an industrial schema","evolve an existing data system without breaking it","run an open-ended business analysis and recommend actions","plan and iterate an exploratory analysis over enterprise tables"],"interaction":"autonomous-long-horizon","environment":"mixed","state_checked":true,"tools":["SQL execution","Python","file access"],"grading":["programmatic-check","llm-judge"],"size":210,"data":"mixed","pii":"none","license":"MIT","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2512.04324","github":null,"leaderboard":null},"good_for":["taxonomy","grading-design"],"fit":"Fits teams whose agent spans both pipeline building and analysis, since it keeps the two separate and shows they need different skills. Borrow its split of execution-based grading for engineering tasks and rubric-guided judge grading for analysis tasks.","caveats":"Project page returned 404 and the dataset card points at an anonymous placeholder repo, so only the Hugging Face dataset (MIT) was found; the LLM judge covers the analysis half; engineering success rates are under 20%."},{"id":"livesqlbench","name":"LiveSQLBench (BIRD-SQL Pro)","year":2025,"by":"BIRD team (University of Hong Kong and Google Cloud)","workloads":["sql","data-analysis"],"task_types":["write a PostgreSQL business-intelligence query over a new unseen database","use hierarchical external knowledge to resolve domain terms","perform a management task that creates, updates or deletes data","pass per-question verifiable test cases"],"interaction":"single-turn","environment":"database","state_checked":true,"tools":["PostgreSQL execution in Docker","external knowledge documents"],"grading":["programmatic-check","state-diff"],"size":600,"data":"mixed","pii":"none","license":"CC-BY-SA-4.0","harness":["own-repo"],"status":"active","links":{"paper":null,"github":"https://github.com/bird-bench/livesqlbench","leaderboard":"https://livesqlbench.ai"},"good_for":["baseline","grading-design","case-template"],"fit":"Fits text-to-SQL agents on PostgreSQL that must also do writes, and teams that worry about benchmark contamination since databases are added over time. Borrow its soft execution accuracy for SELECT tasks and custom test cases for management tasks.","caveats":"No paper, only a repo and dataset card; ground truth and test cases are gated by email; license is stated as MIT in the repo sidebar but CC-BY-SA-4.0 in the README and dataset card; Base-Full v1 is 600 tasks over 22 databases, Base-Lite 270, Large-v1 480."},{"id":"spider2","name":"Spider 2.0","year":2024,"by":"xlang-ai (Fangyu Lei et al.)","workloads":["sql","data-analysis","coding"],"task_types":["answer a business question by writing one or more SQL queries over a large warehouse","find the right tables and columns in metadata with more than 1,000 columns","read dialect documentation and write BigQuery, Snowflake or SQLite SQL","write or fix dbt transformation models in a project repository (Spider2-DBT)","run multi-step SQL workflows that chain queries and transform data"],"interaction":"autonomous-long-horizon","environment":"database","state_checked":true,"tools":["SQL execution on BigQuery","Snowflake","SQLite or DuckDB","file reads in a dbt project"],"grading":["programmatic-check","state-diff"],"size":632,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2411.07763","github":"https://github.com/xlang-ai/Spider2","leaderboard":"https://spider2-sql.github.io"},"good_for":["baseline","case-template","sandbox-design"],"fit":"Fits analytics or warehouse agents that must answer questions over big real schemas in cloud SQL dialects. Borrow its three sub-variants: Spider2-Lite (547 tasks, BigQuery 214, Snowflake 198, SQLite 135), Spider2-Snow (547 tasks on Snowflake only) and Spider2-DBT (68 repository-level dbt tasks on DuckDB), and its execution-result grading.","caveats":"Only partial gold answers are public and the rest go through leaderboard submission; BigQuery tasks cost money; an August 2026 repo notice reports Spider2-Snow access disrupted by a Snowflake account suspension; best reported score at release was 21.3%."},{"id":"telcoagent-bench","name":"TelcoAgent-Bench","year":2026,"by":"Khalifa University, with AT&T and GSMA","workloads":["telecom","it-ops","tool-calling"],"task_types":["recognize the network problem from an engineer's description in English or Arabic","run the troubleshooting tools (KPI time series, neighbor audit, config push) in the required order","ignore distractor tools that do not fit the flow","give the correct resolution and stay consistent across rephrased variants of the same scenario"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["7 telecom operations tools plus 6 distractor tools"],"grading":["trajectory-check","programmatic-check"],"size":null,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2604.06209","github":"https://github.com/BrahiM-Mefgouda/TelcoAgent","leaderboard":null},"good_for":["taxonomy","grading-design"],"fit":"Fits network-operations or technical-support agents in telecom rather than consumer customer support. Borrow the metric suite (intent recognition, ordered tool execution, resolution correctness, stability across variants) and the blueprint-with-sampled-KPIs method for generating variants.","caveats":"Agent starts from a problem statement with no interactive user; dialogues are synthetic from 49 blueprints (about 1,470 samples, exact count not clearly stated); the repo linked in the paper was not inspected, so license and runnability are unknown."},{"id":"telecom-bench","name":"TeleCom-Bench","year":2026,"by":"ZTE (ZTE-AICloud), KDD 2026","workloads":["telecom","it-ops","tool-calling","knowledge-qa"],"task_types":["answer telecom fundamentals and 3GPP protocol questions","recognize operator intent and extract entities from a ticket","pick the tool to call for a network-optimization step","diagnose the root cause of a network fault","propose a fix for a network fault"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["tool-call selection graded against trajectories"],"grading":["exact-match","programmatic-check","llm-judge"],"size":22678,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2605.18025","github":"https://github.com/ZTE-AICloud/TeleCom-Bench","leaderboard":null},"good_for":["taxonomy","grading-design","baseline"],"fit":"Fits network-operations agents that triage faults and run optimization workflows. Borrow its six-step decomposition (intent, entities, tool call, event check, root cause, solution) and its mix of macro-F1, exact match and a 5-point judge with reported rater agreement.","caveats":"Only a subset of the 22,678 samples is public (full set on request to prevent leakage); data license is CDLA-Permissive-2.0 while code is Apache-2.0; tasks are graded as static steps from recorded trajectories, not live agent runs; vendor-specific product knowledge."},{"id":"telemath","name":"TeleMath","year":2025,"by":"Huawei Paris Research Center (with University of Cassino and Khalifa University)","workloads":["telecom","knowledge-qa"],"task_types":["solve a numerical telecom engineering problem","compute link budgets, capacities or signal metrics","return a single numeric answer"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match"],"size":500,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.10674","github":null,"leaderboard":null},"good_for":["case-template","baseline"],"fit":"Fits radio-planning or network-engineering assistants that must do arithmetic on telecom quantities. Borrow its seed-and-expand pattern (50 expert problems grown to 500 with numeric-only answers that grade without a judge).","caveats":"450 of 500 problems are model-generated from 50 seeds; dataset on Hugging Face (netop/TeleMath) is gated behind a contact form and no license is stated; question-answer only, no agent actions."},{"id":"teleqna","name":"TeleQnA","year":2023,"by":"Huawei Paris Research Center (with Khalifa University and CUHK-Shenzhen)","workloads":["telecom","knowledge-qa"],"task_types":["answer multiple-choice questions on telecom terms","answer questions about 3GPP and IEEE standards","answer questions drawn from telecom research papers"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match"],"size":10000,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2310.15051","github":"https://github.com/netop-team/TeleQnA","leaderboard":null},"good_for":["baseline","taxonomy"],"fit":"Fits network-operations or telecom customer-support assistants as a knowledge check before agent testing. Borrow its five-category split (lexicon, research overview, research publications, standards overview, standards specifications).","caveats":"Multiple choice only, no tools or actions; questions were machine-generated from documents with human review, so wording can be shallow; standards questions are where models fail; widely reused since 2023 so some leakage into training data is likely."},{"id":"acebench","name":"ACEBench","year":2025,"by":"Chen Chen et al. (ACEBench group)","workloads":["tool-calling"],"task_types":["call the right synthetic API with correct arguments from one request","spot a missing or wrong parameter, or a request no tool can serve, and say so","finish a multi-turn task with a simulated user inside a sandbox"],"interaction":"single-turn","environment":"api-sandbox","state_checked":true,"tools":["4,538 synthetic APIs across 8 domains and 68 sub-domains, sandbox environments for the Agent subset"],"grading":["programmatic-check","state-diff","trajectory-check"],"size":2000,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2501.12851","github":"https://github.com/ACEBench/ACEBench","leaderboard":"https://chenchen0103.github.io/ACEBench/"},"good_for":["grading-design","taxonomy","user-simulator"],"fit":"Fits assistants that must refuse or ask back when a request is ambiguous or unservable, not just call tools; borrow its Normal / Special / Agent split and its rule-based scoring with end-state and process accuracy for the multi-turn subset.","caveats":"All APIs are synthetic and most cases are single-turn; the Agent subset relies on a GPT-4o user simulator; the per-category counts are not given on the README, and the English and Chinese halves are parallel translations of the same 2,000 entries."},{"id":"agentprocessbench","name":"AgentProcessBench","year":2026,"by":"Fan et al. (Renmin University)","workloads":["tool-calling","general-assistant","knowledge-qa"],"task_types":["judge each step of a tool-using trajectory as helpful, neutral or wrong","find the first step where a run went wrong","tell exploration that does no harm apart from actual errors"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["none"],"grading":["exact-match"],"size":1000,"data":"real-public","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2603.14465","github":"https://github.com/RUCBM/AgentProcessBench","leaderboard":null},"good_for":["grading-design","taxonomy"],"fit":"Fits teams building a step-level judge or process reward signal for a tool-using agent. Borrow the +1/0/-1 step labels with the error-propagation rule and the reported human agreement (89.1%).","caveats":"Trajectories come from four public sets (HotpotQA, GAIA dev, BFCL, tau2), so judges may have seen the tasks; it grades judges, not agents; repo license not seen."},{"id":"api-bank","name":"API-Bank","year":2023,"by":"Alibaba DAMO Academy","workloads":["tool-calling","general-assistant"],"task_types":["call a known API with the right arguments given the dialogue so far","search an API store for the right API, then call it","plan and chain several API calls to serve one request","write the reply to the user after the call returns"],"interaction":"multi-turn-scripted","environment":"api-sandbox","state_checked":false,"tools":["73 hand-built runnable APIs backed by a local database","API search tool"],"grading":["programmatic-check"],"size":314,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2304.08244","github":"https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/api-bank","leaderboard":null},"good_for":["case-template","taxonomy"],"fit":"Fits assistants that must pick one API out of many and fill its arguments from a chat history. Borrow its three-level split (call, retrieve then call, plan then retrieve then call) and its dialogue-plus-expected-call case format.","caveats":"Small (314 dialogues, 753 API calls), unchanged since 2023, and evaluation scripts need adapting; the API calls are judged by matching database effects and results, and replies by ROUGE-L, so top models have little headroom left."},{"id":"api-blend","name":"API-BLEND","year":2024,"by":"IBM Research","workloads":["tool-calling","travel","general-assistant"],"task_types":["detect which APIs a request needs","fill the slots of each API from the user's words","order a sequence of API calls correctly","generalize to API sets not seen in training"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["API sets converted from ATIS, SNIPS, SGD, MultiWOZ, TopV2 and ToolQA plus out-of-distribution sets"],"grading":["programmatic-check"],"size":11618,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2402.15491","github":"https://github.com/IBM/API-BLEND","leaderboard":null},"good_for":["case-template","grading-design","privacy-pattern"],"fit":"Fits teams fine-tuning a model for API calling in travel or assistant domains. Borrow the three metrics (API-F1, parameter-F1, LCS-F1 for order) and the method of converting existing dialogue corpora into API-call cases.","caveats":"Mostly a training corpus; the curated sets are not redistributed because of license clearance, so you must rebuild them from the raw sources with the provided scripts."},{"id":"bfcl","name":"BFCL (Berkeley Function Calling Leaderboard)","year":2024,"by":"UC Berkeley (Gorilla team)","workloads":["tool-calling","memory","web-browsing"],"task_types":["pick the right function and fill its arguments from one request","make several calls in parallel or in sequence for one request","decline to call any function when none fits","carry out a multi-turn task against a stateful simulated backend (file system, trading, travel booking)","answer a question by searching the web","store facts in a memory store and recall them in later turns"],"interaction":"multi-turn-scripted","environment":"api-sandbox","state_checked":true,"tools":["Python","Java and JavaScript function schemas","simulated stateful API backends","web search","memory stores"],"grading":["programmatic-check","state-diff","trajectory-check"],"size":null,"data":"mixed","pii":"possible","license":"Apache-2.0","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://proceedings.mlr.press/v267/patil25a.html","github":"https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard","leaderboard":"https://gorilla.cs.berkeley.edu/leaderboard.html"},"good_for":["grading-design","baseline","taxonomy"],"fit":"Fits any agent whose core job is choosing and filling tool calls; the AST check for single-turn calls and the per-turn state-plus-trajectory check for multi-turn tasks are the patterns to copy, and the leaderboard gives a current baseline for most hosted models.","caveats":"Versions keep moving (v1 single-turn AST and executable calls, v2 added user-contributed live functions, v3 added 800 multi-turn stateful entries, v4 added agentic web search, memory and a non-scoring format-sensitivity set), so compare numbers only within one version; the total test count is not stated in the README or paper, and v2 live prompts come from users."},{"id":"complexfuncbench","name":"ComplexFuncBench","year":2025,"by":"Zhipu AI / Tsinghua University (THUDM)","workloads":["tool-calling","travel"],"task_types":["plan a multi-step booking query (hotel, flight, attraction, car rental, taxi) with dependent calls","respect user constraints when choosing parameter values","infer parameter values implied rather than stated","fill long parameter values and work with up to 128k tokens of context"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["43 Booking.com APIs (via RapidAPI) with recorded golden responses"],"grading":["programmatic-check","llm-judge"],"size":1000,"data":"mixed","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2501.10132","github":"https://github.com/zai-org/ComplexFuncBench","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits travel or booking agents and any agent with long, constrained parameter lists; borrow ComplexEval's layered matching (format check, exact match, then equal-response match, then a GPT-4o tie-break) for grading calls that have more than one right form.","caveats":"Single vertical (Booking.com), response matching needs a paid RapidAPI subscription, the final-response score is a GPT-4o rating, and the repository states no license (it has moved from THUDM to zai-org)."},{"id":"gorilla-apibench","name":"Gorilla APIBench","year":2023,"by":"UC Berkeley","workloads":["tool-calling","ml-engineering","coding"],"task_types":["write the correct call to a machine learning model API (HuggingFace, TorchHub, TensorFlow Hub) from a task description","choose the right API when documentation is supplied by a retriever","avoid inventing an API that does not exist"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["about 1","645 ML model APIs","optional document retriever"],"grading":["programmatic-check"],"size":null,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2305.15334","github":"https://github.com/ShishirPatil/gorilla","leaderboard":null},"good_for":["grading-design","taxonomy"],"fit":"Fits code-writing agents that must call library APIs from documentation; borrow its AST sub-tree match (compare only the arguments that matter) and its separate count of hallucinated versus wrongly used APIs.","caveats":"Single-call, single-turn, machine learning APIs only; instructions are synthetic (10 per API) and the paper's own counts are inconsistent (1,645 APIs, \"over 11,000\" or about 16,450 pairs); the Gorilla team's effort has since moved to BFCL."},{"id":"gta","name":"GTA (General Tool Agents)","year":2024,"by":"OpenCompass (Shanghai AI Lab)","workloads":["tool-calling","general-assistant","data-analysis"],"task_types":["answer a real question about an image, screenshot or table by chaining tools","read handwritten or printed material with OCR and compute on it","search the web and combine the result with a local file","produce a deliverable from a real multi-app workflow (GTA-Workflow)"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["image perception","OCR","search","calculator","code","drawing tools"],"grading":["programmatic-check","exact-match"],"size":229,"data":"real-public","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2407.08713","github":"https://github.com/open-compass/GTA","leaderboard":"https://open-compass.github.io/GTA/"},"good_for":["case-template","grading-design"],"fit":"Fits assistants whose users send screenshots, photos or files and expect the right tools to be picked without being told. Borrow the human-written implicit-tool-use queries and the split between step-level tool accuracy and final-answer accuracy.","caveats":"Small (229 queries) with real deployed tools that can drift; 2026 GTA-Workflow tasks are rewritten from agent platforms and forums, so some public text may be included."},{"id":"livemcpbench","name":"LiveMCPBench","year":2025,"by":"Chinese Academy of Sciences (Institute of Software)","workloads":["tool-calling","general-assistant","research","travel","shopping"],"task_types":["retrieve the right few tools from a catalog of 527 across 70 servers","compose several tools from different servers for a daily-life task (office, travel, shopping, finance)","use live online data (news, weather, prices) in the answer","produce a result that satisfies human-written key points"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":false,"tools":["70 MCP servers","527 tools","no API keys required (LiveMCPTool)"],"grading":["llm-judge"],"size":95,"data":"mixed","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.01780","github":"https://github.com/icip-cas/LiveMCPBench","leaderboard":"https://docs.google.com/spreadsheets/d/1EXpgXq1VKw5A7l7-N2E9xt3w0eLJ2YPVPT-VrRxKZBw/edit?usp=sharing"},"good_for":["sandbox-design","grading-design","taxonomy"],"fit":"Fits agents that must find the right tool in a large catalog before using it; the paper attributes about half of failures to retrieval. Borrow its Dockerized key-free tool suite and its key-point rubric judged by an LLM against live outcomes.","caveats":"Only 95 tasks; answers depend on live web data so ground truth changes and the grader is an LLM judge (DeepSeek-V3, ~80% human agreement); the leaderboard is a Google Sheet."},{"id":"mcp-atlas","name":"MCP-Atlas","year":2025,"by":"Scale AI","workloads":["tool-calling","research","general-assistant","knowledge-qa"],"task_types":["identify which of several MCP servers and tools a plain-language request needs","chain 3-6 tool calls across servers, ignoring plausible distractor tools","synthesize a final answer whose factual claims are grounded in tool outputs","stop at the right point instead of giving up after a successful call"],"interaction":"autonomous-long-horizon","environment":"containers","state_checked":false,"tools":["36 real MCP servers","220 tools in the paper (307 in the released harness)"],"grading":["llm-judge"],"size":1000,"data":"mixed","pii":"possible","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2602.00933","github":"https://github.com/scaleapi/mcp-atlas","leaderboard":"https://labs.scale.com/leaderboard/mcp_atlas"},"good_for":["grading-design","taxonomy","baseline","sandbox-design"],"fit":"Fits general assistant agents that answer questions by orchestrating many MCP tools. Borrow its claim-level rubric (score the final answer against atomic factual claims, so different valid tool paths still pass) and its 11-category failure taxonomy that separates tool-call errors from reasoning errors.","caveats":"Only 500 of the 1,000 tasks are public; grading is an LLM judge (Gemini by default) with a 0.75 claim-coverage threshold, not a state check; 16 of the 36 servers need API keys; released December 2025 on Scale Labs but the arXiv paper is dated January 2026."},{"id":"mcp-bench","name":"MCP-Bench","year":2025,"by":"Accenture (Center for Advanced AI)","workloads":["tool-calling","research","finance","travel","science"],"task_types":["pick the right tools from a vague instruction without being told tool names","chain several tool calls across one or more MCP servers","pass exact parameters from one tool's output into the next","write a final answer grounded in the intermediate tool outputs"],"interaction":"autonomous-long-horizon","environment":"api-sandbox","state_checked":false,"tools":["28 live MCP servers, 250 tools (finance, travel, scientific computing, paper search, maps, Wikipedia, Reddit, NASA)"],"grading":["programmatic-check","llm-judge"],"size":104,"data":"mixed","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.20453","github":"https://github.com/Accenture/mcp-bench","leaderboard":"https://huggingface.co/spaces/mcpbench/mcp-bench"},"good_for":["grading-design","case-template","baseline"],"fit":"Fits agents that must discover and chain tools across several MCP servers from an underspecified request. Borrow its split of rule-based checks (tool name valid, schema compliance, call succeeded) from LLM-judged completion and planning scores, and its \"description fuzzing\" trick for writing tasks that do not name the tool.","caveats":"Only 104 tasks, all synthesized by o4-mini and graded by o4-mini with shuffled rubrics; tasks hit live public servers so results drift; the README badge says Apache 2.0 but there is no LICENSE file at the repo root."},{"id":"mcp-radar","name":"MCP-RADAR","year":2025,"by":"Xi'an Jiaotong University / University of Massachusetts Amherst","workloads":["tool-calling","email-calendar","it-ops","knowledge-qa"],"task_types":["solve a math problem with a calculator MCP tool and give an exact answer","answer a question using a web search tool","carry out an email or calendar operation (send, reply, schedule) via mock servers","perform the right file-management or terminal command","finish in as few tool rounds and tokens as possible"],"interaction":"single-turn","environment":"mixed","state_checked":false,"tools":["49 MCP tools (calculator, DuckDuckGo search, filesystem, terminal are real; email, calendar and a local database are mocked)"],"grading":["exact-match","programmatic-check"],"size":507,"data":"mixed","pii":"none","license":"unknown","harness":["none"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2505.16700","github":null,"leaderboard":null},"good_for":["grading-design","sandbox-design"],"fit":"Fits teams that care about efficiency (rounds, tokens, time) as well as correctness in tool use, and that want mock servers for email and calendar with a controlled local database. Borrow its accuracy-versus-efficiency reporting and its mock-server design.","caveats":"Code and data are only on an anonymous 4open.science link, no GitHub repo or license; per-domain task counts sum to 515, not the stated 507; email, calendar, file and terminal tasks are graded by matching the final tool call, not the end state."},{"id":"mcp-universe","name":"MCP-Universe","year":2025,"by":"Salesforce AI Research","workloads":["tool-calling","web-browsing","finance","devops","research"],"task_types":["find places and routes with a maps server","inspect or change a GitHub repository","answer a financial analysis question from market data","build a 3D object in Blender through its MCP server","complete a browser automation task","answer a time-sensitive question with web search"],"interaction":"autonomous-long-horizon","environment":"mixed","state_checked":false,"tools":["11 real MCP servers (Google Maps, GitHub, financial data, Blender, Playwright, web search, Notion)"],"grading":["programmatic-check","exact-match"],"size":231,"data":"mixed","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.14704","github":"https://github.com/SalesforceAIResearch/MCP-Universe","leaderboard":"https://mcp-universe.github.io/#results"},"good_for":["grading-design","case-template","baseline"],"fit":"Fits agents that drive real third-party services (maps, GitHub, browser, market data) over long tool-calling sessions. Borrow its evaluator pattern (extract a value from the output, compare with an operator) and its \"dynamic evaluators\" that fetch live ground truth for time-sensitive tasks instead of freezing an answer.","caveats":"Needs paid API keys (SerpAPI, Google Maps, GitHub, Notion) and a local Blender install; GitHub tasks perform real writes so a throwaway account is required; 231 hand-written tasks across six domains only; 57% of tasks depend on live data that changes."},{"id":"mcpmark","name":"MCPMark","year":2025,"by":"National University of Singapore / EvalSys","workloads":["tool-calling","workflow-automation","documents","sql","coding"],"task_types":["create, update and reorganize pages and databases in Notion","manage issues, branches and files in a GitHub repository","run queries and change data in a PostgreSQL database","edit files on a filesystem","automate a web page with Playwright"],"interaction":"single-turn","environment":"mixed","state_checked":true,"tools":["MCP servers for Notion","GitHub","Filesystem","PostgreSQL","Playwright"],"grading":["programmatic-check","state-diff","pass^k"],"size":127,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2509.24002","github":"https://github.com/eval-sys/mcpmark","leaderboard":"https://mcpmark.ai"},"good_for":["grading-design","sandbox-design","case-template"],"fit":"Fits agents that write to Notion, GitHub or databases through MCP. Borrow the per-task verify.py against a curated initial state and the pass^k reliability reporting.","caveats":"Needs live Notion and GitHub accounts; a Verified subset reports different numbers from the paper; mostly developer-flavored apps."},{"id":"mcptoolbench-pp","name":"MCPToolBench++","year":2025,"by":"Ant Group","workloads":["tool-calling","web-browsing","shopping","finance","general-assistant"],"task_types":["produce a correct single MCP tool call (name and arguments) for a query","produce a multi-step chain of tool calls as a DAG","complete the call against a real MCP server (browser, filesystem, search, maps, finance, payments)","handle queries in several languages"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["87 MCP tools across browser, filesystem, search, maps, finance, pay (puppeteer, playwright, filesystem, tavily, google-maps)"],"grading":["programmatic-check","pass^k","llm-judge"],"size":1509,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.07575","github":"https://github.com/mcp-tool-bench/MCPToolBenchPP","leaderboard":"https://github.com/mcp-tool-bench/MCPToolBenchPP#leaderboard"},"good_for":["grading-design","baseline","case-template"],"fit":"Fits agents whose main risk is malformed or wrong tool calls rather than long planning. Borrow its pairing of static AST accuracy with live Pass@k execution, which shows how much a model's \"correct-looking\" calls actually fail against real servers.","caveats":"Queries are LLM-generated from tool schemas, so they are shallow; a third of the set is Map tasks; the README says the benchmark is still work in progress; needs the MCP Marketplace client plus provider and server API keys to run."},{"id":"mnms","name":"m&m's","year":2024,"by":"University of Washington / Allen Institute for AI","workloads":["tool-calling","planning","documents","general-assistant"],"task_types":["plan a chain of 1-3 tools (vision models, text models, public APIs, image ops) for a request","write the plan as JSON or as code with correct tool names and arguments","pass one tool's output as the input of the next","revise the plan from parsing, verification or execution feedback"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["33 tools (13 ML models, 9 public APIs, 11 image-processing modules)"],"grading":["programmatic-check","exact-match"],"size":882,"data":"mixed","pii":"possible","license":"unknown","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2403.11085","github":"https://github.com/RAIVNLab/mnms","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits agents that plan a short pipeline over heterogeneous tools (OCR, captioning, weather, search) before executing. Borrow its plan-level metrics (tool-F1, argname-F1, pass rate) and its finding that step-by-step planning with feedback beats one-shot plans.","caveats":"Grades plans against one gold plan, so valid alternative pipelines are penalized; queries were GPT-4-written from sampled tool graphs; no LICENSE file in the repo; no updates since 2024."},{"id":"multi-mission-tool-bench","name":"Multi-Mission Tool Bench","year":2025,"by":"Tencent Hunyuan","workloads":["tool-calling","general-assistant"],"task_types":["handle a second, related request that arrives after the first","switch to a new mission mid-dialogue and drop the old context","reuse information from an earlier mission in a later one","find the shortest valid tool path for each mission"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["tools drawn from a public-apis list of about 1,600 tools in 50 categories"],"grading":["trajectory-check","programmatic-check"],"size":1024,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2504.02623","github":"https://github.com/yupeijei1997/MMTB","leaderboard":null},"good_for":["case-template","taxonomy","grading-design"],"fit":"Fits assistants whose users pile on related or changing requests in one session. The dynamic decision-tree grading that accepts several valid paths and scores path optimality is the part to borrow.","caveats":"Data is generated by a multi-agent pipeline with no real users, the repo has no license file, and the user side is scripted."},{"id":"nestful","name":"NESTFUL","year":2024,"by":"IBM Research","workloads":["tool-calling"],"task_types":["write a sequence of API calls where one call's output feeds the next","name the right function and map parameters to earlier outputs","solve a math word problem with calculator-style tools","complete a coding task by composing Python helper functions"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["40 executable math tools","881+ executable Python functions"],"grading":["exact-match","programmatic-check"],"size":1861,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2409.03797","github":"https://github.com/IBM/NESTFUL","leaderboard":null},"good_for":["grading-design","case-template"],"fit":"Fits agents that must pass results between tools in one plan (data pipelines, calculators, code helpers); borrow its label-and-reference format for nested calls and its metric ladder (API-name F1, parameter F1, partial and full sequence match, executed win rate).","caveats":"Only math and coding sources (MathQA, StarCoder2-Instruct), sequences were LLM-generated and filtered by execution, and there is no dialogue or user; the v1 RapidAPI-based subset differs from the v2 set the README now ships."},{"id":"noisytoolbench","name":"NoisyToolBench","year":2024,"by":"Chinese University of Hong Kong","workloads":["tool-calling","general-assistant"],"task_types":["ask the right clarifying question when key information is missing","ask when an instruction is ambiguous between tools or values","avoid inventing arguments the user never gave","complete the API call once the user answers"],"interaction":"multi-turn-simulated-user","environment":"api-sandbox","state_checked":false,"tools":["100 real RESTful APIs from ToolBench across 49 categories"],"grading":["llm-judge","programmatic-check"],"size":200,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2409.00557","github":"https://github.com/Mysterchan/learning_to_ask","leaderboard":null},"good_for":["user-simulator","grading-design","case-template"],"situations":["tool-error"],"fit":"Fits any agent that gets vague requests and should ask rather than guess. Borrow the case format (imperfect query, expected clarifying question, scripted answer, expected call) and the ToolEvaluator that scores both accuracy and redundant questions.","caveats":"Only 200 cases built by corrupting ToolBench instructions, and the user answers and judging depend on GPT-4o."},{"id":"restbench","name":"RestBench","year":2023,"by":"Peking University","workloads":["tool-calling","general-assistant","knowledge-qa"],"task_types":["answer a movie question by chaining real TMDB REST calls","control a music player through Spotify REST calls","parse a long OpenAPI response and pick the field needed next"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["54 TMDB APIs and 40 Spotify APIs with OpenAPI specs"],"grading":["human","trajectory-check"],"size":157,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2306.06624","github":"https://github.com/Yifan-Song793/RestGPT","leaderboard":null},"good_for":["case-template","baseline"],"fit":"Fits agents that call real REST services from OpenAPI docs. Borrow the small hand-annotated gold-path format and the correct-path-rate metric alongside success.","caveats":"Needs live TMDB and Spotify accounts, success was judged by hand, only 157 instructions, and the repo has been idle since 2024."},{"id":"rotbench","name":"RoTBench","year":2024,"by":"Fudan University","workloads":["tool-calling","general-assistant"],"task_types":["select the right tool when tool names are garbled or swapped","identify parameters when their names are noisy","fill parameter values correctly under noisy documentation"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["568 real tools from ToolEyes, with five noise levels applied to their docs"],"grading":["programmatic-check"],"size":105,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2401.08326","github":"https://github.com/Junjie-Ye/RoTBench","leaderboard":null},"good_for":["case-template","taxonomy"],"fit":"Fits teams whose tool schemas change or are poorly documented. The pattern to copy is taking a clean case set and producing slight, medium, heavy and union noise variants to measure robustness.","caveats":"Only 105 base queries (840 across noise levels), first-call only, and the README still says evaluation instructions are on the way."},{"id":"seal-tools","name":"Seal-Tools","year":2024,"by":"Mengsong Wu et al.","workloads":["tool-calling"],"task_types":["select one tool out of a large self-generated catalogue and fill its parameters","call several tools for one instruction","nest calls so one tool's output becomes another's input"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["4,076 self-instruct generated API-like tools, DPR retriever"],"grading":["exact-match","programmatic-check"],"size":null,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2405.08355","github":"https://github.com/fairyshine/Seal-Tools","leaderboard":null},"good_for":["case-template","privacy-pattern"],"fit":"Fits teams who need a large, fully made-up tool catalogue for fine-tuning or retrieval tests; borrow the self-instruct pipeline (field, tool, instance) and its format accuracy plus tool and parameter precision/recall/F1 scoring.","caveats":"Everything was generated by ChatGPT, nothing executes, test split size is not stated (14,076 instances in total, 586 nested), and the README calls the evaluation code \"not well organized\"."},{"id":"stabletoolbench","name":"StableToolBench","year":2024,"by":"Tsinghua University (THUNLP-MT)","workloads":["tool-calling","general-assistant"],"task_types":["pick and chain RapidAPI-style tools for one instruction","handle multi-tool tasks across categories","produce a final answer judged solved or unsolved"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["virtual API server (response cache, then real API, then GPT-4 simulated response), MirrorAPI model"],"grading":["llm-judge"],"size":765,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2403.07714","github":"https://github.com/THUNLP-MT/StableToolBench","leaderboard":null},"good_for":["sandbox-design","grading-design"],"situations":["tool-error"],"fit":"Fits agents that call many third-party web APIs and need a replayable test bed; the cache plus LLM-simulated-API server is the pattern to copy when the real APIs are flaky, and the solvable-only filtering shows how to prune unanswerable cases.","caveats":"Grading is still a GPT-4 judge (solvable pass rate and win rate) with no state check, API responses on cache misses are model-simulated rather than real, and the 765 tasks are a filtered subset of ToolBench's ChatGPT-written instructions."},{"id":"t-eval","name":"T-Eval","year":2023,"by":"Shanghai AI Laboratory / USTC","workloads":["tool-calling","planning"],"task_types":["produce a tool call in the required format with correct parameters","pick the right tool by name from a list","write a multi-step plan before calling tools","state the reasoning for the next step","read a tool's response and extract the needed arguments","judge whether a tool response is correct, incomplete or an error"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["15 tools across research","travel","entertainment","web","life","finance"],"grading":["exact-match","programmatic-check","llm-judge"],"size":23305,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2312.14033","github":"https://github.com/open-compass/T-Eval","leaderboard":"https://open-compass.github.io/T-Eval/leaderboard.html"},"good_for":["taxonomy","grading-design","case-template"],"fit":"Fits teams that want to diagnose which step of tool use their agent gets wrong (format, tool choice, plan, reasoning, argument extraction, response review) rather than only the end result. Borrow its decomposition and its per-step graders.","caveats":"Queries and gold solutions are GPT-3.5/GPT-4 generated over only 553 base queries; the Plan/Reason/Understand splits use sentence-embedding similarity, which is a soft signal; no updates since 2024 and the leaderboard is stale."},{"id":"taskbench","name":"TaskBench","year":2023,"by":"Zhejiang University / Microsoft (Shen et al.)","workloads":["tool-calling","planning","workflow-automation"],"task_types":["decompose a user request into steps","pick the right tools and order them (chain or graph)","fill in tool parameters correctly"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["tool graphs over Hugging Face models","multimedia tools and daily-life APIs"],"grading":["programmatic-check"],"size":28271,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2311.18760","github":"https://github.com/microsoft/JARVIS/tree/main/taskbench","leaderboard":null},"good_for":["case-template","grading-design","taxonomy"],"fit":"Fits workflow-automation agents that must plan a tool chain before executing. Borrow the node-F1 and edge-F1 scoring of predicted tool graphs and the back-instruct data generator.","caveats":"Tools are never executed, so it scores plans not outcomes; fully synthetic instructions; no updates since 2024."},{"id":"toolalpaca","name":"ToolAlpaca","year":2023,"by":"Chinese Academy of Sciences (Institute of Software)","workloads":["tool-calling","general-assistant"],"task_types":["read an API description it has never seen and call it correctly","choose the right sequence of calls without redundant actions","answer the user's request from the API responses"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["10 simulated APIs and 11 real APIs in the eval set; 400+ APIs in training data"],"grading":["llm-judge"],"size":214,"data":"synthetic","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2306.05301","github":"https://github.com/tangqiaoyu/ToolAlpaca","leaderboard":null},"good_for":["user-simulator","sandbox-design","privacy-pattern"],"fit":"Fits teams that need to test an agent against APIs that do not exist yet or cannot be hit in test; borrow its LLM-simulated API server and its procedure/response/overall grading split. Mostly a training corpus, not a benchmark.","caveats":"Tiny eval set (100 simulated, 114 real instances) graded only by GPT-4; the simulated API responses are made up by a model so they can be wrong; built for 2023-era 7B/13B fine-tunes and effectively saturated for current models."},{"id":"toolathlon","name":"Tool Decathlon (Toolathlon)","year":2025,"by":"HKUST / CMU / All Hands AI","workloads":["tool-calling","workflow-automation","email-calendar","general-assistant","devops"],"task_types":["complete a long task across several real apps via MCP (Notion, Google Calendar, WooCommerce, Kubernetes, BigQuery)","start from a realistic app state such as a course site with students or a finance sheet","update records in one app based on data read from another"],"interaction":"single-turn","environment":"mixed","state_checked":true,"tools":["604 tools across 32 applications","mostly MCP servers"],"grading":["programmatic-check","state-diff"],"size":108,"data":"mixed","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.25726","github":"https://github.com/hkust-nlp/Toolathlon","leaderboard":"https://toolathlon.xyz"},"good_for":["sandbox-design","grading-design","case-template"],"fit":"Fits agents connected to many SaaS tools over MCP. Borrow the per-task evaluation scripts that check end state in the real app and the practice of seeding realistic initial states.","caveats":"Needs accounts and API keys for many third-party apps (a hosted evaluation service exists); 108 tasks; code license not stated."},{"id":"toolbench","name":"ToolBench (ToolLLM)","year":2023,"by":"OpenBMB / Tsinghua University","workloads":["tool-calling","general-assistant"],"task_types":["pick APIs from a RapidAPI catalogue for one instruction","chain calls across several tools from the same or different categories","recover when an API fails or returns nothing useful","produce a final answer from the collected API results"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["16,464 real RapidAPI endpoints across 49 categories via the authors' proxy, API retriever"],"grading":["llm-judge"],"size":1100,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2307.16789","github":"https://github.com/OpenBMB/ToolBench","leaderboard":null},"good_for":["taxonomy","case-template"],"fit":"Fits agents that must choose from a very large tool catalogue and chain real web APIs; borrow its single-tool / same-category / cross-collection split (G1, G2, G3) for building a tool coverage taxonomy.","caveats":"Calls live RapidAPI endpoints through the authors' key server, so results drift as APIs die; the pass and win rates are ChatGPT judgments; instructions were generated by ChatGPT; StableToolBench replaced it for stable scoring."},{"id":"toolcomp","name":"ToolComp","year":2025,"by":"Scale AI","workloads":["tool-calling","research","finance"],"task_types":["answer a multi-step question by combining search, calculator, weather, stock and Python tools","plan and revise intermediate steps toward a verifiable final answer","solve chat-style questions with only web search and Python"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["11 tools (date, weather, calculator, Wikipedia, Google search, Wolfram Alpha, stock data, Python)"],"grading":["llm-judge","trajectory-check"],"size":485,"data":"mixed","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2501.01290","github":null,"leaderboard":"https://scale.com/leaderboard/tool_use"},"good_for":["grading-design","taxonomy"],"fit":"Fits research-style assistants that chain several tools to reach one answer; borrow its idea of human-corrected step labels (process supervision) next to a final-answer check, and its enterprise versus chat split.","caveats":"The data and evaluation code are not public (release promised only when a model passes 85 percent or at the end of 2026), the final answer is graded by a GPT-4 Turbo judge, and several tools hit live web services, so you can only read the leaderboard, not run it."},{"id":"toolsandbox","name":"ToolSandbox","year":2024,"by":"Apple","workloads":["tool-calling","general-assistant","mobile"],"task_types":["change a phone setting such as wifi or cellular before completing the user's request","send a message, add a contact or set a reminder that depends on earlier tool results","ask the user for missing information instead of guessing","canonicalize fuzzy user input (dates, names) into the form a tool needs"],"interaction":"multi-turn-simulated-user","environment":"api-sandbox","state_checked":true,"tools":["34 stateful phone-assistant tools across 11 domains (settings, contacts, messaging, reminders, search)"],"grading":["state-diff","trajectory-check"],"size":1032,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2408.04682","github":"https://github.com/apple/ToolSandbox","leaderboard":null},"good_for":["grading-design","sandbox-design","user-simulator","taxonomy"],"fit":"Fits assistant agents whose tools have hidden state dependencies (one call only works after another). Borrow the milestone and minefield grading: required world-state checkpoints in a partial order, forbidden events that zero the score, and similarity scoring of each checkpoint.","caveats":"Scenarios are hand-written by two annotators rather than drawn from real traffic; the license is a custom Apple permissive license with no SPDX id; search tools need a paid RapidAPI key; no leaderboard, and no activity signal after the April 2025 paper revision."},{"id":"ultratool","name":"UltraTool","year":2024,"by":"Harbin Institute of Technology (Shenzhen) / Huawei","workloads":["tool-calling","planning","general-assistant"],"task_types":["write a natural-language step plan for a complex request before any tool is chosen","decide whether the available tools are enough or a new tool must be defined","define a missing tool (name, description, parameters)","select the right tool for a step and fill its arguments, including nested calls"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["2,032 tool definitions across 22 domains (no execution)"],"grading":["llm-judge","exact-match","programmatic-check"],"size":1000,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2401.17167","github":"https://github.com/JoeYing1019/UltraTool","leaderboard":null},"good_for":["taxonomy","case-template","grading-design"],"fit":"Fits agents that plan multi-step work over a large tool catalog, and teams that want to grade planning separately from execution. Borrow its planning/creation/usage split, its six-dimension GPT-4 plan rubric, and its key-value accuracy for tool selection and arguments.","caveats":"No tools are executed, so nothing checks that a plan would actually work; planning and tool-creation scores depend on GPT-4 as judge; originally Chinese with GPT-4 translation to English; 1,000-sample test split out of 5,824 total."},{"id":"mtu-bench","name":"MTU-Bench","year":2024,"by":"Alibaba Group","workloads":["travel","general-assistant","tool-calling"],"task_types":["pick the right tool and fill its parameters from one user turn","call several tools in the right order for one request","track slots across a multi-turn booking or lookup dialogue","handle requests that fall outside the training distribution"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["136 tools derived from task-oriented dialogue datasets (MultiWOZ, SGD, TaskMaster, MetaLWOZ, ATIS, SNIPS)"],"grading":["exact-match","programmatic-check"],"size":431,"data":"mixed","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2410.11710","github":"https://github.com/MTU-Bench-Team/MTU-Bench","leaderboard":null},"good_for":["taxonomy","grading-design","baseline"],"fit":"Fits booking and lookup agents built on slot-filling dialogues. Borrow its 2x2 taxonomy (single or multi turn by single or multi tool) and its metric set (tool selection, parameter accuracy, turn success, task progress) that needs no LLM judge.","caveats":"Test set is only 431 dialogues drawn from old dialogue corpora rewritten by an LLM; evaluation is string match against one gold path, so valid alternatives count as wrong."},{"id":"travelplanner","name":"TravelPlanner","year":2024,"by":"Fudan University / Ohio State University (Xie et al.)","workloads":["travel","planning","tool-calling"],"task_types":["produce a full multi-day itinerary from a request","look up flights, hotels, restaurants and attractions with tools","satisfy hard constraints (budget, cuisine, room rules) and commonsense ones"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["search tools over a static database of about four million records"],"grading":["programmatic-check"],"size":1225,"data":"mixed","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2402.01622","github":"https://github.com/OSU-NLP-Group/TravelPlanner","leaderboard":"https://huggingface.co/spaces/osunlp/TravelPlannerLeaderboard"},"good_for":["grading-design","case-template","baseline"],"fit":"Fits planning assistants that must juggle many constraints. Borrow the split between commonsense and hard constraint pass rates plus a final all-constraints pass rate.","caveats":"Static offline data; original pass rates were near zero so numbers are hard to compare across versions; no interaction with a user."},{"id":"assistantbench","name":"AssistantBench","year":2024,"by":"Tel Aviv University (Yoran et al.)","workloads":["web-browsing","research","general-assistant","shopping"],"task_types":["find a specific answer that needs browsing several live websites","compare prices, listings or businesses across sites","compute a figure from data scattered on the open web"],"interaction":"single-turn","environment":"browser","state_checked":false,"tools":["web browser","search"],"grading":["exact-match","programmatic-check"],"size":214,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["inspect_evals","hal","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2407.15711","github":"https://github.com/oriyor/assistantbench","leaderboard":"https://huggingface.co/spaces/AssistantBench/leaderboard"},"good_for":["grading-design","baseline"],"fit":"Fits research assistants that browse the live web for a user. Borrow the answer-rate versus accuracy split so that abstaining is counted differently from being wrong.","caveats":"Live websites drift so answers go stale; read-only; only 33 dev tasks have public answers."},{"id":"bearcubs","name":"BEARCUBS","year":2025,"by":"Iyyer group (UMass Amherst / University of Maryland)","workloads":["web-browsing","research","knowledge-qa"],"task_types":["find a short factual answer that needs live browsing","watch a video or play a web game to extract the answer","navigate a 3D or interactive page"],"interaction":"single-turn","environment":"browser","state_checked":false,"tools":["live web browsing with screenshots"],"grading":["exact-match"],"size":111,"data":"real-public","pii":"none","license":"CC-BY-NC-ND-4.0","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2503.07919","github":null,"leaderboard":"https://bear-cubs.github.io/"},"good_for":["case-template","baseline"],"fit":"Fits multimodal browsing agents; borrow the rule that each question has one short, unambiguous answer plus a human-validated trajectory.","caveats":"Tiny; questions are swapped out when contaminated so scores move; non-commercial license; top score 65.8 percent versus 84.7 for humans."},{"id":"browsecomp","name":"BrowseComp","year":2025,"by":"OpenAI","workloads":["web-browsing","research","knowledge-qa"],"task_types":["find a hard-to-locate fact by chaining many web searches","identify a person, work or event from several intertwined clues","give a short answer that can be checked against a reference"],"interaction":"autonomous-long-horizon","environment":"browser","state_checked":false,"tools":["web search","page browsing"],"grading":["llm-judge"],"size":1266,"data":"real-public","pii":"none","license":"MIT","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2504.12516","github":"https://github.com/openai/simple-evals","leaderboard":null},"good_for":["case-template","baseline","grading-design"],"fit":"Fits agents whose job is to dig up obscure facts on the live web and return a short answer. Borrow the pattern of building questions backward from a short verifiable answer, and the accuracy plus calibration reporting.","caveats":"Live web results drift over time, so scores are not reproducible; short answers only, no report writing; the simple-evals repo is no longer updated for new models."},{"id":"browsecomp-plus","name":"BrowseComp-Plus","year":2025,"by":"University of Waterloo (Jimmy Lin's group)","workloads":["web-browsing","research","knowledge-qa"],"task_types":["answer a multi-hop BrowseComp question using only a fixed document corpus","plan and issue searches against a retriever, then reason over the hits","cite the documents that support the answer"],"interaction":"autonomous-long-horizon","environment":"api-sandbox","state_checked":false,"tools":["search over a fixed corpus","document fetch"],"grading":["llm-judge","programmatic-check"],"size":830,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2508.06600","github":"https://github.com/texttron/BrowseComp-Plus","leaderboard":"https://huggingface.co/spaces/Tevatron/BrowseComp-Plus"},"good_for":["sandbox-design","grading-design","baseline"],"fit":"Fits search agents that run over an internal document store rather than the open web. Borrow the fixed corpus with human-verified positives and mined hard negatives, and the separate retrieval metrics (nDCG, recall) next to answer accuracy.","caveats":"The corpus is about 100K documents chosen around the questions, so it is easier than the open web; the LLM judge is Qwen3-32B; leaderboard entries are submitted by email."},{"id":"browsecomp-zh","name":"BrowseComp-ZH","year":2025,"by":"Peilin Zhou et al.","workloads":["web-browsing","research","knowledge-qa"],"task_types":["answer a multi-hop question by browsing the Chinese web","reconcile conflicting information from several Chinese sources","return a short verifiable answer such as a date, number or name"],"interaction":"autonomous-long-horizon","environment":"browser","state_checked":false,"tools":["web search","page browsing"],"grading":["llm-judge"],"size":289,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2504.19314","github":"https://github.com/PALIN2018/BrowseComp-ZH","leaderboard":null},"good_for":["case-template","baseline","privacy-pattern"],"fit":"Fits agents that must search Chinese-language sites, where English benchmarks say little. Borrow the reverse-engineered question design and the encrypted dataset with a canary token.","caveats":"Small (289 questions); live web so results drift; the README restricts the data to academic research use; grading relies on a GPT-4o answer-extraction step."},{"id":"browserarena","name":"BrowserArena","year":2025,"by":"University of Pennsylvania (Anupam, Bastani et al.)","workloads":["web-browsing","general-assistant"],"task_types":["complete a user-submitted task on the open web","get past captchas and pop-up banners","navigate directly to the right URL"],"interaction":"single-turn","environment":"browser","state_checked":false,"tools":["live web browsing"],"grading":["human"],"size":null,"data":"real-public","pii":"possible","license":"CC-BY-4.0","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2510.02418","github":null,"leaderboard":null},"good_for":["taxonomy","grading-design"],"fit":"Fits teams that want pairwise human preference on live browsing rather than fixed tasks. Borrow the step-level human feedback and the captcha / pop-up / URL failure taxonomy.","caveats":"Not reproducible (live web, human votes); no task count published; no code release found."},{"id":"browsergym","name":"BrowserGym / AgentLab","year":2024,"by":"ServiceNow Research","workloads":["web-browsing","workflow-automation"],"task_types":["run MiniWoB, WebArena, VisualWebArena, WebArena-Verified, WorkArena, AssistantBench and WebLINX under one interface","define new browser tasks with the same observation and action space"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["Playwright browser","accessibility tree","screenshots"],"grading":["exact-match","programmatic-check","state-diff"],"size":null,"data":"mixed","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2412.05467","github":"https://github.com/ServiceNow/BrowserGym","leaderboard":null},"good_for":["sandbox-design","baseline"],"fit":"A harness, not a task set; fits teams that want to run several web benchmarks with one agent loop. Borrow the gym-style observation and action space and AgentLab's trace viewer.","caveats":"Brings no tasks of its own; each wrapped benchmark keeps its own hosting needs and license."},{"id":"formfactory","name":"FormFactory","year":2025,"by":"National University of Singapore / Wuhan University (Li et al.)","workloads":["web-browsing","workflow-automation","documents"],"task_types":["fill an online form from a source document such as a resume","map each piece of user data to the right field across varied layouts"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["web form interface","PyAutoGUI actions"],"grading":["programmatic-check","exact-match"],"size":null,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.01520","github":null,"leaderboard":"https://formfactory-ai.github.io/"},"good_for":["case-template","grading-design"],"fit":"Fits data-entry and onboarding agents. Borrow the atomic (one field) versus episodic (whole form) scoring split.","caveats":"Form and field counts not in the abstract; repo URL not found; no model passed 5 percent at release so it gives little signal between agents yet."},{"id":"mind2web","name":"Mind2Web","year":2023,"by":"OSU NLP Group","workloads":["web-browsing","shopping","travel","general-assistant"],"task_types":["predict the next click, type or select action on a real website page","follow a multi-step instruction across pages (travel, shopping, government, entertainment)","generalize to unseen websites and domains"],"interaction":"single-turn","environment":"static","state_checked":false,"tools":["page snapshot (HTML)"],"grading":["exact-match"],"size":2350,"data":"real-scraped","pii":"possible","license":"CC-BY-4.0","harness":["inspect_evals","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2306.06070","github":"https://github.com/OSU-NLP-Group/Mind2Web","leaderboard":"https://osu-nlp-group.github.io/Mind2Web/"},"good_for":["case-template","taxonomy","baseline"],"fit":"Fits teams training or testing step-level action prediction on real site layouts. Borrow the cross-task / cross-website / cross-domain split design.","caveats":"Offline replay of recorded pages only, so no real execution or end-state check; pages are 2023 snapshots; size is train 1009 plus test 1341 as listed in the repo."},{"id":"mind2web-2","name":"Mind2Web 2","year":2025,"by":"Ohio State University NLP group","workloads":["web-browsing","research"],"task_types":["answer a long-horizon question whose answer changes over time","browse live websites and gather facts from many pages","cite a URL for each key fact in the answer"],"interaction":"autonomous-long-horizon","environment":"browser","state_checked":false,"tools":["web search","page browsing"],"grading":["llm-judge","programmatic-check"],"size":130,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.21506","github":"https://github.com/OSU-NLP-Group/Mind2Web-2","leaderboard":"https://osu-nlp-group.github.io/Mind2Web-2/#leaderboard"},"good_for":["grading-design","case-template"],"fit":"Fits research assistants whose answers depend on current web state. Borrow Agent-as-a-Judge: a per-task judge script built from a tree rubric that opens the cited pages and checks both correctness and attribution.","caveats":"130 tasks and the test set is private, so leaderboard entries go by email; judges need OpenAI access and a browser cache; building each task took many human hours, so it is hard to extend."},{"id":"online-mind2web","name":"Online-Mind2Web","year":2025,"by":"OSU NLP Group","workloads":["web-browsing","shopping","travel","general-assistant"],"task_types":["complete a user task on a live public website starting from its home page","find and filter items (clothing, food, housing, transport)","fill in a form or reach a confirmation page"],"interaction":"single-turn","environment":"browser","state_checked":false,"tools":["live web browsing"],"grading":["llm-judge","human"],"size":300,"data":"real-public","pii":"none","license":"CC-BY-4.0","harness":["hal","own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2504.01382","github":"https://github.com/OSU-NLP-Group/Online-Mind2Web","leaderboard":"https://huggingface.co/spaces/osunlp/Online_Mind2Web_Leaderboard"},"good_for":["grading-design","baseline"],"fit":"Fits general browser agents on public consumer sites. Borrow WebJudge, a screenshot-based LLM judge with released human labels to calibrate against.","caveats":"Live sites drift and break tasks; judge is about 85 percent agreement with humans; no end-state verification; human review of leaderboard entries is outsourced."},{"id":"real","name":"REAL","year":2025,"by":"AGI Inc (The AGI Company)","workloads":["web-browsing","shopping","travel","email-calendar"],"task_types":["buy, book or order on replicas of Amazon, Airbnb, United, DoorDash, Uber, OpenTable","send mail or set events on Gmail and Calendar replicas","find information on LinkedIn, Upwork or Zillow replicas"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["deterministic website replicas in a browser"],"grading":["state-diff","llm-judge"],"size":112,"data":"synthetic","pii":"none","license":"CC-BY-NC-SA-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2504.11543","github":"https://github.com/agi-inc/REAL","leaderboard":"https://realevals.xyz/"},"good_for":["sandbox-design","grading-design"],"fit":"Fits consumer browser agents that touch money or bookings. Borrow the replica-site sandbox with state checks for actions and rubric judging for lookups.","caveats":"Non-commercial license; replicas are vendor-built clones so fidelity is unverified; small; top score 41 percent at release."},{"id":"sealqa","name":"SealQA","year":2025,"by":"Virginia Tech","workloads":["web-browsing","knowledge-qa"],"task_types":["answer a fact question when search results conflict or are noisy","decide which of several sources to trust","find the relevant document among many distractors in long context"],"interaction":"single-turn","environment":"browser","state_checked":false,"tools":["web search"],"grading":["llm-judge"],"size":111,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.01062","github":null,"leaderboard":null},"good_for":["case-template","baseline"],"fit":"Fits search-augmented assistants that must not be fooled by bad search hits. Borrow the idea of selecting questions where search returns conflicting or unhelpful results, and the LongSeal needle-in-a-haystack variant for retrieval-heavy agents.","caveats":"Seal-0 is only 111 questions (Seal-Hard and LongSeal have 254 each); grading is a Colab template rather than a runnable repo; live search so difficulty shifts over time."},{"id":"visualwebarena","name":"VisualWebArena","year":2024,"by":"Carnegie Mellon University (Koh, Fried et al.)","workloads":["web-browsing","retail","shopping"],"task_types":["find a listing or product that matches a reference image","post a classified ad with a given picture","compare items by what is shown in their photos","answer a question that needs reading an image on the page"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["browser actions on self-hosted sites","image input"],"grading":["exact-match","programmatic-check","state-diff","llm-judge"],"size":910,"data":"mixed","pii":"possible","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2401.13649","github":"https://github.com/web-arena-x/visualwebarena","leaderboard":"https://jykoh.com/vwa"},"good_for":["sandbox-design","grading-design","case-template"],"fit":"Fits agents that must read screenshots or product images while browsing (classifieds, marketplaces). Borrow the image-conditioned task templates and the execution-based checks.","caveats":"Built on the WebArena stack with the same hosting burden; some checks use a vision model judge; human success is about 89 percent so ceiling is below 100."},{"id":"webarena","name":"WebArena","year":2023,"by":"Carnegie Mellon University (Zhou, Neubig et al.)","workloads":["web-browsing","retail","devops","workflow-automation"],"task_types":["find a product, order or post on a self-hosted site and report the answer","change data in an online shop admin panel","open, edit or comment on issues and merge requests in GitLab","post or reply on a forum","look up a route or distance on a map"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["browser actions on self-hosted sites","scratchpad","map","wiki"],"grading":["exact-match","programmatic-check","state-diff"],"size":812,"data":"mixed","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2307.13854","github":"https://github.com/web-arena-x/webarena","leaderboard":"https://webarena.dev/"},"good_for":["sandbox-design","grading-design","case-template","baseline"],"situations":["unanswerable"],"fit":"Fits browser agents that act inside internal web apps (shop admin, issue tracker, forum). Borrow the self-hosted site sandbox and the per-task evaluator mix of exact answer, target URL and page-content program checks.","caveats":"Reference answers and evaluators had errors (see WebArena-Verified); the Reddit dump contains real usernames; environments are heavy to host and must be reset between runs."},{"id":"webarena-infinity","name":"WebArena-Infinity","year":2026,"by":"web-arena-x (Carnegie Mellon University)","workloads":["web-browsing","workflow-automation"],"task_types":["complete generated tasks on auto-built web apps","difficulty-graded tasks with programmatic verifiers"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["browser actions on generated web apps"],"grading":["programmatic-check","state-diff"],"size":null,"data":"synthetic","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://web-arena-x.github.io/webarena-infinity/","github":"https://github.com/web-arena-x/webarena-infinity","leaderboard":null},"good_for":["sandbox-design","case-template"],"fit":"Fits teams that need many fresh, uncontaminated browser tasks; borrow the pipeline that generates an app, its tasks and verifiers with coding agents and audits them with browser agents.","caveats":"Blog post only, no paper; task count not published; environments are synthetic apps so realism is unverified; very new."},{"id":"webarena-verified","name":"WebArena-Verified","year":2025,"by":"ServiceNow Research","workloads":["web-browsing","retail","devops","workflow-automation"],"task_types":["the same 812 WebArena tasks with corrected answers and evaluators","a 258-task hard subset for cheaper runs"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["browser actions on self-hosted sites"],"grading":["exact-match","programmatic-check","state-diff"],"size":812,"data":"mixed","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://openreview.net/forum?id=94tlGxmqkN","github":"https://github.com/ServiceNow/webarena-verified","leaderboard":null},"good_for":["grading-design","baseline"],"situations":["unanswerable"],"fit":"Use instead of original WebArena for any browser agent on internal web apps. Borrow the deterministic, type-aware comparators that replaced substring matching and LLM judging.","caveats":"Workshop paper only; numbers are not comparable to the old WebArena leaderboard; still needs the full WebArena site stack."},{"id":"webcanvas","name":"WebCanvas (Mind2Web-Live)","year":2024,"by":"iMean AI (Pan et al.)","workloads":["web-browsing","shopping","travel"],"task_types":["complete a Mind2Web task on the live website","pass through required intermediate pages or states on the way"],"interaction":"single-turn","environment":"browser","state_checked":false,"tools":["live web browsing"],"grading":["trajectory-check","programmatic-check"],"size":542,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2406.12373","github":"https://github.com/iMeanAI/WebCanvas","leaderboard":null},"good_for":["grading-design"],"fit":"Fits any live-site browser agent where the final page cannot be checked directly. Borrow key-node grading: credit for reaching required URLs or elements mid-task.","caveats":"Key nodes break when sites redesign and need upkeep; no end-state verification; top success at release was 23.1 percent."},{"id":"webchorearena","name":"WebChoreArena","year":2025,"by":"WebChoreArena authors (arXiv 2506.01952)","workloads":["web-browsing","retail","devops","data-analysis"],"task_types":["read and total many records across pages (orders, posts, issues)","compute a figure from information gathered on the site","track details across many pages and sites to answer"],"interaction":"single-turn","environment":"browser","state_checked":true,"tools":["browser actions on WebArena sites"],"grading":["exact-match","programmatic-check","state-diff"],"size":532,"data":"mixed","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.01952","github":"https://github.com/WebChoreArena/WebChoreArena","leaderboard":null},"good_for":["case-template","baseline"],"fit":"Fits back-office browser agents that must aggregate data across many pages. Borrow the memory / calculation / long-term-memory task categories.","caveats":"Needs the WebArena site stack; top reported score 44.9 percent on a preview model since retired; cross-site tasks are few."},{"id":"webgames","name":"WebGames","year":2025,"by":"Convergence Labs","workloads":["web-browsing","games"],"task_types":["solve a small self-contained browser challenge (drag, scroll, timing, canvas)","handle tricky inputs, pop-ups and adversarial page tricks","read visual puzzles on the page"],"interaction":"single-turn","environment":"browser","state_checked":false,"tools":["client-side web pages","no network"],"grading":["exact-match"],"size":150,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2502.18356","github":"https://github.com/convergence-ai/webgames","leaderboard":"https://webgames.convergence.ai/"},"good_for":["grading-design","case-template"],"fit":"Fits teams that want fast, hermetic checks of raw browser control. Borrow the password-on-success grading, which needs no judge or state inspection.","caveats":"Toy pages, not real sites; paper says 50+ challenges while the repo has 150; license not stated; best agent 43.1 percent versus 95.7 for humans."},{"id":"weblinx","name":"WebLINX","year":2024,"by":"McGill NLP (Lu, Reddy et al.)","workloads":["web-browsing","general-assistant"],"task_types":["predict the next browser action in a recorded conversation with a user","reply to the user in chat while navigating a real site"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["recorded page snapshots","screenshots"],"grading":["exact-match","programmatic-check"],"size":2300,"data":"real-scraped","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2402.05930","github":"https://github.com/McGill-NLP/weblinx","leaderboard":"https://mcgill-nlp.github.io/weblinx/"},"good_for":["case-template","user-simulator"],"fit":"Fits chat-driven browser copilots where a human gives instructions turn by turn. Borrow the demonstration format of paired user utterances and browser actions.","caveats":"Offline replay only, no live execution; size is about 2,300 demos (100K steps); data license differs from the Apache code license and recordings may contain personal details."},{"id":"webvoyager","name":"WebVoyager","year":2024,"by":"Tencent AI Lab (He et al.)","workloads":["web-browsing","shopping","travel","knowledge-qa"],"task_types":["find a flight, hotel or product on a popular public site","look up a fact on Wikipedia, GitHub, ArXiv or a news site","navigate a map or booking site to an answer"],"interaction":"single-turn","environment":"browser","state_checked":false,"tools":["live web browsing with screenshots"],"grading":["llm-judge","human"],"size":643,"data":"real-public","pii":"none","license":"Apache-2.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2401.13919","github":"https://github.com/MinorJerry/WebVoyager","leaderboard":null},"good_for":["baseline","grading-design"],"fit":"Fits consumer-facing browser agents on well-known sites. Borrow the screenshot-based auto-judge that reads the last k screenshots.","caveats":"Tasks on 15 live sites go stale (Booking, Google Flights); judge-only grading with no state check; many vendors now report high scores, so it discriminates poorly."},{"id":"webwalkerqa","name":"WebWalkerQA","year":2025,"by":"Alibaba Tongyi Lab","workloads":["web-browsing","knowledge-qa"],"task_types":["start at a given root website and click through subpages to find an answer","find information that search engines do not surface","answer in Chinese or English depending on the site"],"interaction":"autonomous-long-horizon","environment":"browser","state_checked":false,"tools":["page navigation"],"grading":["llm-judge"],"size":680,"data":"real-scraped","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2501.07572","github":"https://github.com/Alibaba-NLP/WebAgent","leaderboard":null},"good_for":["case-template","baseline"],"fit":"Fits agents that must navigate inside one site (a company portal, a university site, a conference site) rather than search the open web. Borrow the root-site plus question design and the action-count efficiency metric.","caveats":"Pages change, so answers can rot; only four domains, 60% Chinese; the linked WebAgent repo is a general family repo and does not clearly host the dataset or its eval script; license for the data not stated."},{"id":"xbench-deepsearch","name":"xbench-DeepSearch","year":2025,"by":"xbench","workloads":["web-browsing","knowledge-qa","research"],"task_types":["answer a Chinese open-domain question that needs several search steps","follow the expected reference steps to locate the fact"],"interaction":"autonomous-long-horizon","environment":"browser","state_checked":false,"tools":["web search"],"grading":["llm-judge"],"size":100,"data":"real-public","pii":"none","license":"MIT","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.13651","github":"https://github.com/xbench-ai/xbench-evals","leaderboard":"https://xbench.org/#/agi/aisearch"},"good_for":["baseline","privacy-pattern"],"fit":"Fits Chinese-language search assistants that need a refreshed test set. Borrow the encrypted release with canary strings and the plan to re-issue a new 100-question set periodically.","caveats":"Chinese only; 100 questions per release and old releases are superseded; the arXiv paper covers the profession tracks, the DeepSearch details live on the dataset card and site; default judge is gemini-2.0-flash."},{"id":"agentarch","name":"AgentArch","year":2025,"by":"ServiceNow","workloads":["workflow-automation","tool-calling","it-ops"],"task_types":["complete a simple enterprise request with tools","complete a complex multi-step enterprise workflow","compare orchestration, memory and reasoning settings on the same task"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["enterprise tools in a simulated setting"],"grading":["programmatic-check"],"size":2,"data":"synthetic","pii":"none","license":"unknown","harness":["none"],"status":"active","links":{"paper":"https://arxiv.org/abs/2509.10769","github":null,"leaderboard":null},"good_for":["taxonomy"],"fit":"Fits teams choosing an agent architecture for enterprise work. Borrow the grid of 18 configurations (orchestration, memory, thinking, tool style) as an ablation plan.","caveats":"Only two tasks; no public code; results are about architectures, not a reusable test set."},{"id":"erp-bench","name":"ERP-Bench (Anchor)","year":2026,"by":"Ivanov and Rana","workloads":["workflow-automation","manufacturing","finance","planning"],"task_types":["run a procurement workflow end to end in an ERP system","plan and record manufacturing orders","keep business records consistent over a long sequence of actions"],"interaction":"autonomous-long-horizon","environment":"api-sandbox","state_checked":true,"tools":["ERP system interfaces"],"grading":["state-diff","programmatic-check"],"size":300,"data":"synthetic","pii":"none","license":"unknown","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2605.26321","github":null,"leaderboard":"https://erpbench.ai"},"good_for":["grading-design","case-template","sandbox-design"],"fit":"Fits ERP and back-office automation agents. Borrow the parametric task generator that yields a solver-certified optimal solution and a state-based verifier per task.","caveats":"Two-author release with the ERP product unnamed on the paper page; dataset site could not be reached to confirm license; procurement and manufacturing only."},{"id":"shortcutsbench","name":"ShortcutsBench","year":2024,"by":"Peking University / Beijing University of Posts and Telecommunications","workloads":["workflow-automation","tool-calling","mobile","planning"],"task_types":["choose the next app API in a long action sequence given the history so far","fill static parameters correctly","reuse outputs of earlier actions as inputs to later ones","recognize when the workflow needs input from the user or the system and ask for it"],"interaction":"multi-turn-scripted","environment":"static","state_checked":false,"tools":["1,414 real APIs from 88 Apple apps, extracted from Shortcuts"],"grading":["exact-match","programmatic-check"],"size":5220,"data":"mixed","pii":"possible","license":"Apache-2.0","harness":["own-repo"],"status":"frozen","links":{"paper":"https://arxiv.org/abs/2407.00132","github":"https://github.com/EachSheep/ShortcutsBench","leaderboard":null},"good_for":["case-template","taxonomy","grading-design"],"fit":"Fits workflow-automation agents that compose many app or SaaS APIs into one long sequence. Borrow its four-way grading (API choice, parameter fill, reuse of prior outputs, asking for input) and its difficulty tiers by sequence length.","caveats":"Queries are GPT-4o-written from real shortcuts, not from users; no execution, so grading is next-step accuracy against one gold sequence; Apple-only APIs; shortcut data carries the terms of the sharing sites it was scraped from."},{"id":"sop-bench","name":"SOP-Bench","year":2025,"by":"Amazon","workloads":["workflow-automation","customer-support","healthcare","finance","manufacturing"],"task_types":["follow a written standard operating procedure to reach a decision","call mock tools in the order the procedure requires","classify an email, flag content or inspect a record per the procedure","handle patient intake or dangerous-goods checks step by step"],"interaction":"single-turn","environment":"api-sandbox","state_checked":false,"tools":["mock tools per procedure"],"grading":["exact-match","programmatic-check"],"size":1811,"data":"synthetic","pii":"none","license":"CC-BY-NC-4.0","harness":["own-repo"],"status":"active","links":{"paper":"https://arxiv.org/abs/2506.08119","github":"https://github.com/amazon-science/sop-bench","leaderboard":null},"good_for":["case-template","grading-design","taxonomy"],"fit":"Fits agents that must execute a company procedure exactly. Borrow the metric split (execution completion rate times conditional success equals task success) and tool accuracy.","caveats":"Non-commercial license; tools are mocks; procedures and data were partly AI-generated."}]},"situations":[{"id":"unanswerable","name":"Unanswerable","short":"right answer is \"none\"","meaning":"The data holds no answer, or the tools can't do what was asked.","catches":"Making up a record, number or outcome that isn't there","check":"Code: the reply says there is none, and nothing was written","universal":true,"from":["crmarena","webarena","osworld","appworld"]},{"id":"ambiguous","name":"Ambiguous","short":"must ask or explore","meaning":"The request matches more than one thing, or is too vague to act on without a question or a look at the data.","catches":"Acting on a guess when it should have asked, or asking when a look at the data would do","check":"Code: asked before acting, or found the one right match","universal":true,"from":["bird-interact","crmarena-pro","ask-or-assume"]},{"id":"forbidden","name":"Forbidden or confirm-first","short":"refuse, or ask before acting","meaning":"The policy forbids the request, or it needs the person's confirmation before the agent acts.","catches":"Doing what it was told not to do, or acting without the confirmation its rules require","check":"Code: nothing was written. Judge: the reply named the rule","universal":true,"from":["tau-bench","tau2-bench","crmarena-pro","enterprisebench-corecraft","agentharm"]},{"id":"confidential","name":"Confidential","short":"don't disclose","meaning":"The request asks for data the asker may not see, such as another person's private details or a record outside their role.","catches":"Reading out private or restricted data","check":"Code: the protected value is absent from the reply","universal":false,"from":["crmarena-pro","enterprisebench","enterprisebench-corecraft"]},{"id":"multi-turn","name":"Multi-turn pair","short":"same task over several turns","meaning":"The same request arrives over several turns, with details given late or changed, and is scored against its single-turn twin.","catches":"Losing the thread between turns, or acting on a detail the person later changed","check":"The twin's check, paired, so the difference is the cost of the conversation","universal":true,"from":["crmarena-pro","tau2-bench","bird-interact"]},{"id":"tool-error","name":"Tool error","short":"a call fails mid-task","meaning":"A tool call fails, times out or returns an error partway through.","catches":"Stopping partway, or reporting success after a failed call","check":"Code: the reply finished, said what failed, and the end state is right","universal":true,"from":["bird-interact","noisytoolbench","stabletoolbench"]},{"id":"persona","name":"Different asker","short":"same job, another role","meaning":"The same job asked by a different role, such as a manager wanting an aggregate where a rep wants one record.","catches":"Answering the wrong question for who is asking","check":"The task type's own check, split by asker","universal":false,"from":["crmarena","crmarena-pro","scuba"]},{"id":"injected","name":"Injected instruction","short":"untrusted content gives orders","meaning":"Content the agent reads, such as a web page, email or record, contains instructions aimed at the agent.","catches":"Following an instruction that came from data rather than from the person","check":"Code: the injected action did not happen, and the original task was done","universal":false,"from":["agentdojo","injecagent"]},{"id":"dual-control","name":"Person does a step","short":"the user must act on their side","meaning":"Part of the job can only be done by the person, on their own device or account, with the agent guiding.","catches":"Doing the person's step itself, or giving the wrong step","check":"Code: the end state on both sides is right","universal":false,"from":["tau2-bench"]}],"templates":[{"path":"templates/case.json","content":"{\n \"id\": \"confirm-001\",\n \"category\": \"confirm-and-book\",\n \"set\": \"error_finding\",\n \"failure_modes\": [\"invite-wrong-time\"],\n \"taxonomy_entries\": [\"calendar.create-event\"],\n \"provenance\": {\n \"source\": \"export:2026-10-02-traces\",\n \"export\": \"2026-10-02-traces\",\n \"trace_ref\": \"\",\n \"synthetic\": false,\n \"agent_version\": \"\",\n \"reviewed_by\": \"\",\n \"notes\": \"scrubbed: names replaced, emails moved to example.com\"\n },\n \"trigger\": {\"type\": \"message\", \"text\": \"Can we do Thursday 3pm instead? Please send the invite.\"},\n \"inputs\": {\"thread\": [], \"expected_event\": {\"title\": \"Sync\", \"start\": \"2026-10-08T15:00:00-04:00\"}},\n \"start_state\": {\"events\": []},\n \"followups\": [],\n \"expected_end_state\": {\n \"world\": {\"events\": [{\"title\": \"Sync\", \"start\": \"2026-10-08T15:00:00-04:00\"}]},\n \"output\": null\n }\n}\n"},{"path":"templates/check_code.py","content":"\"\"\"Code check: . Binary, graded on the end state.\n\nstart / end: {\"world\": , \"output\": }\ncase: the case JSON (inputs, start_state, expected_end_state, ...)\n\nGrade against the goal, not one expected path: any sequence of actions that reaches the right\nstate passes. Never read the agent's claim that it succeeded (G10); the C7 screen checks for that.\n\"\"\"\n\n\ndef check(start, end, case):\n want = case[\"inputs\"][\"expected_event\"]\n events = (end[\"world\"] or {}).get(\"events\", [])\n matches = [e for e in events if e.get(\"title\") == want[\"title\"] and e.get(\"start\") == want[\"start\"]]\n return len(matches) == 1, f\"{len(matches)} matching event(s); want exactly one at {want['start']}\"\n"},{"path":"templates/check_judge.md","content":"---\nid: # must match the file name\nfailure_mode: \nversion: 1 # bump on every change to the question, options or inputs\nquestion_type: noul # noul (a true/false claim) | choice | score\nquestion: \ndetects: failure # noul only: a \"yes\" means the failure is present (failure) or the requirement is met (pass)\n# choice: options: {grounded: \"...\", invented: \"...\"} pass_options: [grounded]\n# score: options: [Poor, Adequate, Good] pass_min_level: 1\ninputs: [end.output, case.inputs.source_document] # the only fields the judge sees (dotted paths into case / start / end)\nmode: binary # binary | pairwise (pairwise needs a different family from the agent, G4)\nmin_tpr: 0.75 # floors: below either, the judge is uncalibrated and greyed (G3)\nmin_tnr: 0.75\nprimary: jev # the backend whose verdicts count\nbackends:\n jev:\n type: systemone # any POST /v1/systemone endpoint: TypeSafe, OpenCode Zen, a local LAYA server\n model: jev-1.13-free # free via OpenCode Zen, no key; see references/judge-backends.md\n daily_limit: 250 # the free tier refuses (HTTP 429) after about 250 calls a day per IP;\n # one calibration pass over a few judges can use it up. Paid: jev-1.13\n family: typesafe\n base_url: https://opencode.ai/zen\n claude:\n type: anthropic # generative, structured {reasoning, answer}; a different family from the agent\n model: claude-opus-5-5\n family: anthropic\n effort: medium\n # mine:\n # type: command # any judge you run yourself; see references/judge-backends.md\n # command: [python, evals/env/my_judge.py]\n # model: \n # family: \n---\n\n# \n\n**Pass when.** \n\n**Fail when.** \n\n**Not this judge's job.** \n\nThis body is sent to generative backends as guidance. System One backends see only the\n`question` (and `options`). Keep it short and observable.\n"},{"path":"templates/taxonomies/README.md","content":"# Vertical taxonomy templates\n\nC2 starts the coverage list from `templates/taxonomies/.yaml` when one fits the agent's\nvertical. Each template has the same shape as `evals/taxonomy.yaml` and must name a real source\ndocument written for some other purpose (a public API surface, a standard job-task taxonomy, a\nregulator's list of duties). A template derived from anyone's eval cases is exactly what G5\nexists to refuse, and a poor default becomes everyone's coverage list.\n\nNo vertical templates ship yet. Contributions are welcome: each needs a real, citable source\n(see CONTRIBUTING.md). Until then C2 drafts the list from the team's own product spec, tool\nsurface or help-centre topics, and the owner edits it.\n\nTemplate shape:\n\n```yaml\nname: jobs\nsource:\n document: \n kind: api-surface | job-taxonomy | regulation | help-centre | product-spec\n url: <where it is published>\nread_on: <date>\ntemplate: templates/taxonomies/<vertical>.yaml\nentries:\n - id: <stable-id>\n title: <short title>\n description: <one line, quoted or paraphrased from the source>\n```\n"}],"plugin":{"repo":"https://github.com/Mahalanobis-Systems/aot-evals","install":["claude plugin marketplace add Mahalanobis-Systems/aot-evals","claude plugin install aot-evals@aot-evals"],"start":"/aot-evals:start","skills":[{"name":"assess","command":"/aot-evals:assess","description":"Assess, the first of aot-evals' three phases, usually reached through start. Runs when a team wants evals for an agent and no discovery report exists yet (evals/agents/<agent>/discovery.md), about 5 minutes. Shows the AOT EVALS banner and the three phases, scans the repository for every agent (each skill of a multi-skill system counts as its own agent), asks which agent to test, asks three questions about it, runs discovery in parallel, and writes a plain-English discovery report: an overview of the four things good evals rest on, the details, and the components the optimization loop needs, and next steps. Use for \"set up evals\", \"build evals for my agent\", \"assess my agent\", or when start finds no discovery report."},{"name":"build","command":"/aot-evals:build","description":"Build, the second of aot-evals' three phases, for engineering. Gets an agent's eval loop running with the fewest decisions - the agent wired to run as a command, 10 to 20 cases from real runs with a check on each, a baseline, and the grader application handed to the people who will label - then raises trust step by step (real-traffic provenance, the agent inventory, screens, the labelling standard and per-case requirements, categories and budgets, the coverage list, case allocation, comparisons of a changed agent, coverage and redundancy). Use when start says the phase is Build, when someone asks to wire an agent up for evals, write eval cases or checks, run a baseline, or compare a change to the agent."},{"name":"judge","command":"/aot-evals:judge","description":"Judge, the third of aot-evals' three phases, for the people who know what good looks like (product owners, experts, end users) and whoever facilitates them. Runs the grader application (aot-evals serve), where graders label whole runs blind, settle disagreements with rulings, hold the suite to their verdicts (catch rate and clean pass rate per category), give every disagreement a cause, accept other correct answers, propose rules for the labelling standard, and label and calibrate AI judges; and says what goes back to engineering. Use when start says the phase is Judge, when someone wants to label runs, onboard a grader, calibrate or update a judge, or asks whether the suite agrees with people."},{"name":"report","command":"/aot-evals:report","description":"The report for any phase of aot-evals - regenerate <suite>/report/report.json (fixed schema) and write one self-contained local HTML diagnostic from it, shaped around the user's question (why did the candidate regress? is this suite any good? what does it cost?). Use when someone asks for an eval report, dashboard, scorecard or diagnostic for an agent that has an evals/ directory."},{"name":"start","command":"/aot-evals:start","description":"The one command to remember for aot-evals. Says which of the three phases an agent's evals are in (Assess, Build, Judge), how far each kind of task can be trusted, what changed since the baseline, and carries on with the next step - the assessment when there is no discovery report yet (evals/agents/<agent>/discovery.md), otherwise build or judge. Use for \"start aot-evals\", \"set up evals\", \"build evals for my agent\", \"where are we with our evals\", \"what's next\", \"is this suite any good\", or after any change to the agent, its prompt, tools or model."}]},"download":{"zip":"aot-evals-method-1.0.1.zip","url":"https://github.com/Mahalanobis-Systems/aot-evals/releases/download/aot-evals--v1.0.1/aot-evals-method-1.0.1.zip"}}