Mahal Systems BioProBench
Fits lab-assistant or protocol-writing agents that must respect procedural order, quantities and safety steps. Borrow the five task shapes (QA, ordering, error correction, generation, reasoning) and the step recall and Kendall tau metrics for ordered procedures.
What it asks the agent to do
- answer a question about a wet-lab protocol
- put shuffled protocol steps in the right order
- find and fix an error in a protocol
- write a full protocol from a goal
- reason about why a protocol step is needed
What to borrow from it
case templatetaxonomygrading design
Caveats
Text-only, no robot or lab environment; non-commercial license; the huge instance count is auto-derived from 22k protocols, so quality is uneven and the held-out test size is not stated in the README; generation is scored with BLEU and keyword overlap.
Details
| Workloads | science, knowledge qa, planning |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | exact match, programmatic check |
| Size | 523,784 |
| Data | real scraped |
| Personal data | none |
| License | CC-BY-NC-4.0 |
| Harness | own repo |
| Status | active |