Mahal Systems KramaBench
Fits analytics agents that work over many raw files rather than one tidy database. Borrow its sub-task decomposition (633 sub-tasks under 104 tasks), its mix of exact metrics, generated unit tests and rubric partial credit, and its six domain split.
What it asks the agent to do
- find the relevant files among about 1,700 raw files in a data lake
- clean and join heterogeneous files from several sources
- write Python that computes a numeric or text answer to a domain question
- break a question into sub-tasks and solve each one
What to borrow from it
grading designtaxonomycase template
Caveats
No license file in the repo and the dataset card only says "cc"; headline accuracy differs between paper versions (55% versus 50%); scientific domains only (archaeology, astronomy, biomedical, environment, legal, wildfire); top repo score 22%.
Details
| Workloads | data analysis, science, coding, research |
|---|---|
| Interaction | single turn |
| Environment | filesystem |
| End state checked | No |
| Tools | Python code execution over local files |
| Grading | programmatic check, unit tests, llm judge |
| Size | 104 |
| Data | real public |
| Personal data | none |
| License | unknown |
| Harness | own repo |
| Status | active |