Mahal Systems BixBench
Fits bioinformatics or data-analysis agents that work inside a notebook against real data. Borrow the "capsule" pattern (a published notebook plus its data turned into open questions) and the Docker notebook sandbox.
What it asks the agent to do
- explore a biological dataset in a notebook
- run a multi-step analysis in Python, R or Bash
- answer an open-ended research question from the results
- answer the same question in multiple-choice form
What to borrow from it
sandbox designgrading designcase template
Caveats
Only 60 source notebooks and 205 questions; open answers are graded by an LLM; the Hugging Face dataset needs authentication; multiple-choice scores were at chance for frontier models at release, so that mode is a weak signal.
Details
| Workloads | science, data analysis, coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | No |
| Tools | Jupyter notebook, Python, R, Bash |
| Grading | llm judge, exact match |
| Size | 205 |
| Data | real public |
| Personal data | none |
| License | Apache-2.0 |
| Harness | inspect_evals, own repo |
| Status | active |