Mahal Systems AstaBench
Fits scientific research-assistant agents that combine literature search, code execution and data analysis. Borrow its reproducible search environment with date cutoffs, its cost-per-task reporting next to accuracy, and its four-category macro average.
What it asks the agent to do
- find papers and answer literature questions with a search tool
- build a literature comparison table
- reproduce results from a paper's code
- run a data-driven discovery analysis
- carry out an end-to-end research task from question to report
What to borrow from it
sandbox designbaselinetaxonomy
Caveats
It is a suite of 11 existing and new benchmarks (CORE-Bench-Hard, DiscoveryBench, DS-1000 and others), so results overlap with those; needs a large machine (authors suggest 128 GB RAM), API keys for scoring, and an Asta tool key for the search tool; the leaderboard is curated by AI2.
Details
| Workloads | science, research, coding, data analysis, planning |
|---|---|
| Interaction | autonomous long horizon |
| Environment | mixed |
| End state checked | No |
| Tools | literature search MCP, Python sandbox, report and table editors |
| Grading | programmatic check, llm judge |
| Size | 2,400 |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |