Mahal Systems LAB-Bench
Fits research-assistant agents for biology that answer questions over papers, databases and sequences. Borrow the eight-category taxonomy (LitQA2, DbQA, SuppQA, FigQA, TableQA, ProtocolQA, SeqQA, Cloning) and the accuracy-plus-coverage reporting that rewards abstaining.
What it asks the agent to do
- answer a literature question that needs the paper's full text
- look up a value in a biological database
- read a figure or table from a paper
- troubleshoot a lab protocol
- manipulate a DNA or protein sequence
- plan a molecular cloning scenario
What to borrow from it
taxonomybaselinecase template
Caveats
Multiple-choice only, so not an agent benchmark on its own; 20 percent of the data is held private; frontier models have largely caught up and the authors released LABBench2 (arXiv 2604.09554, about 1,900 tasks, CC-BY-SA-4.0, github.com/EdisonScientific/labbench2, also in inspect_evals) as the harder successor.
Details
| Workloads | science, knowledge qa, research |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none required; agents may use search or code |
| Grading | exact match |
| Size | 2,400 |
| Data | real public |
| Personal data | none |
| License | CC-BY-SA-4.0 |
| Harness | inspect_evals, own repo |
| Status | active |