Mahal Systems BLADE
Fits research or analytics agents whose output is an analysis plan, not a single number. Borrow the way expert analysts' decisions (constructs, transforms, model) are collected as ground truth and matched to the agent's choices.
What it asks the agent to do
- pick the variables and transforms needed to answer a research question from a dataset
- write a data transform function
- specify a statistical model and justify the choice
What to borrow from it
grading designcase template
Caveats
Only 12 dataset and question pairs; evaluation needs an LLM; the README still has placeholder counts and says a hold-out test set is in progress.
Details
| Workloads | science, data analysis |
|---|---|
| Interaction | single turn |
| Environment | terminal |
| End state checked | No |
| Tools | python code execution |
| Grading | llm judge, programmatic check |
| Size | 12 |
| Data | real public |
| Personal data | none |
| License | ODC-By-1.0 |
| Harness | own repo |
| Status | active |