Mahal Systems ScienceAgentBench
Fits data-analysis or scientific coding agents that take a dataset and a goal and must hand back a runnable program. Borrow its per-task evaluation scripts that execute the generated code and compare outputs, plus the best-of-three-runs scoring.
What it asks the agent to do
- write a self-contained Python program for a data-analysis task from a paper
- load and process a scientific dataset
- produce a figure or model output that matches the expected result
- debug the program when it fails to run
What to borrow from it
grading designcase templatebaseline
Caveats
Only 102 tasks from 44 papers in four fields; figure outputs are judged by GPT-4o so an OpenAI key is required; a verified split was released in 2026, so older numbers do not compare. Code is MIT but a few tasks keep the original data licenses.
Details
| Workloads | science, data analysis, coding, research |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | No |
| Tools | Python, shell |
| Grading | programmatic check, llm judge |
| Size | 102 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo, hal |
| Status | active |