Mahal Systems AutoDCWorkflow benchmark
Fits data cleaning or data preparation agents that take a messy table and a goal. Borrow its three-way grading: does the clean table answer the purpose, how close are the cell values to the gold table, and how close is the operation sequence to a human workflow.
What it asks the agent to do
- given a raw table and an analysis purpose, pick the columns that need cleaning
- fix format inconsistencies, type errors and duplicates with OpenRefine operations
- produce a minimal clean table that answers the stated purpose
- output a cleaning workflow that matches a human-written one
What to borrow from it
grading designcase template
Caveats
No code repository was found; small (142 purposes over 96 tables, six topics); only six OpenRefine operations are covered; baselines are small open models; tables come from public datasets in social science, public health, finance and transportation.
Details
| Workloads | data analysis, workflow automation |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | Yes |
| Tools | OpenRefine operations |
| Grading | programmatic check, exact match |
| Size | 142 |
| Data | real public |
| Personal data | possible |
| License | unknown |
| Harness | none |
| Status | active |