InnovatorBench

Fits ML-engineering or research agents that run multi-hour training experiments. Borrow the ResearchGym pattern (asynchronous monitoring, snapshots, distributed execution) and the task-specific evaluation scripts that score a runnable artifact on a held-out metric.

What it asks the agent to do

• construct or filter a training dataset for a language model
• design a loss or reward function and train with it
• build an agent scaffold and run it
• run long experiments on GPUs and report the result

What to borrow from it

sandbox designgrading design

Caveats

Only 20 tasks from 14 papers; each run needs multiple GPUs and 2 to 36 hours, so it is expensive to repeat; the leaderboard was marked under construction at the time of reading.

Details

Links

• Paper
• Github