Mahal Systems DA-Code
Fits data-analysis or notebook agents that write and run code over files. Borrow its three task categories (data wrangling, machine learning, exploratory analysis), the Docker sandbox and the evaluate.py comparison of output files to gold files.
What it asks the agent to do
- load, clean and transform data files with Python or SQL (data wrangling)
- run exploratory analysis, statistics and visualization on a dataset
- train a classifier, regressor or clustering model and write predictions
- plan multi-step code in a sandbox and write result files to a path
What to borrow from it
sandbox designcase templatetaxonomy
Caveats
Leaderboard only lists 2024 models (best 30.5% with GPT-4) and has not been updated; the full task set is a separate download with only 100 examples in the repo; grading metrics are not described in the README.
Details
| Workloads | data analysis, coding, ml engineering |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | No |
| Tools | Python, SQL and bash in a Docker sandbox |
| Grading | programmatic check |
| Size | 500 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | frozen |