Mahal Systems DataSciBench
Fits agents that take vague analysis requests rather than crisp questions. Borrow the Task-Function-Code idea: each prompt is tied to named aggregate functions and programmatic rules that score the execution result, plus a semi-automatic way to build ground truth.
What it asks the agent to do
- carry out an open-ended data science prompt that has no single right answer
- produce code whose outputs are scored by named metrics
- clean, analyse and visualise a dataset end to end
What to borrow from it
grading designcase template
Caveats
No task count on the abstract page or README, no license file in the repo, and ground truth is partly produced by LLM self-consistency, so it inherits model bias.
Details
| Workloads | data analysis, coding, science |
|---|---|
| Interaction | single turn |
| Environment | terminal |
| End state checked | No |
| Tools | python code execution |
| Grading | programmatic check |
| Size | Not published |
| Data | mixed |
| Personal data | none |
| License | unknown |
| Harness | own repo |
| Status | active |