Mahal Systems DSBench
Fits data science agents that must read long briefs and large or multi-table files, and agents that build models end to end. Borrow the relative performance gap metric that scores a model against the human leaderboard on the same Kaggle task.
What it asks the agent to do
- answer a numeric or multiple-choice question from a workbook with several tables
- read a task brief that includes images and tables before analysing data files
- train a model on a Kaggle dataset and submit predictions
What to borrow from it
case templatebaselinegrading design
Caveats
The analysis answer checker needs an OpenAI key, so grading is model-assisted; the README says the dataset is for non-commercial use even though the code is MIT; the modeling part has only 74 tasks.
Details
| Workloads | data analysis, ml engineering, coding |
|---|---|
| Interaction | single turn |
| Environment | terminal |
| End state checked | No |
| Tools | python notebooks, data files |
| Grading | llm judge, programmatic check |
| Size | 540 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |