Mahal Systems DSEval
Fits notebook-style assistants where each question builds on the last. Borrow the problem set format (ordered problems sharing one session) and the bootstrapped annotation method for expanding cases.
What it asks the agent to do
- solve a sequence of dependent data science steps in one session
- answer a StackOverflow-style pandas question with code
- work through a Kaggle notebook as a chain of problems
What to borrow from it
case templategrading design
Caveats
Four sub-benchmarks of uneven size and difficulty; the README does not describe the validators; sources include LeetCode and StackOverflow, which models have seen.
Details
| Workloads | data analysis, coding |
|---|---|
| Interaction | multi turn scripted |
| Environment | terminal |
| End state checked | No |
| Tools | python code execution |
| Grading | programmatic check |
| Size | 825 |
| Data | mixed |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |