Mahal Systems DSAEval
Fits full data science agents that produce notebooks and reports over several requests. Borrow the three-part judge rubric (reasoning, code, results) with a consistency flag that checks the narrative against what the code produced.
What it asks the agent to do
- carry out a multi-query data science project on one dataset
- analyse unstructured data such as images or text as well as tables
- deliver a notebook and a written report
What to borrow from it
grading designtaxonomy
Caveats
Grading is entirely by two LLM judges against a synthesized soft ground truth; no code repository or license was found; the paper says data will be released gradually.
Details
| Workloads | data analysis, ml engineering, science |
|---|---|
| Interaction | multi turn scripted |
| Environment | terminal |
| End state checked | No |
| Tools | python code execution, image viewing |
| Grading | llm judge |
| Size | 641 |
| Data | real public |
| Personal data | none |
| License | unknown |
| Harness | none |
| Status | active |