Terminal-Bench 2.0

The standard number for terminal and devops agents; frontier agents were under 65 percent at release. Borrow Harbor's container-per-task layout and the human-written oracle solution.

What it asks the agent to do

• complete a hard real-world task in a Linux shell such as fixing a build, setting up a service or recovering data
• write and run code or scripts from the command line until a hidden test passes
• manage long multi-step work in a container without a user

What to borrow from it

baselinesandbox designgrading designcase template

Caveats

Only 89 tasks so confidence intervals are wide; the project has since moved to newer versions (the site now shows 4.0) and a 2.1 release could not be confirmed, so pin the version.

Details

Links

• Paper
• Github
• Leaderboard