Mahal Systems AgentProcessBench
Fits teams building a step-level judge or process reward signal for a tool-using agent. Borrow the +1/0/-1 step labels with the error-propagation rule and the reported human agreement (89.1%).
What it asks the agent to do
- judge each step of a tool-using trajectory as helpful, neutral or wrong
- find the first step where a run went wrong
- tell exploration that does no harm apart from actual errors
What to borrow from it
grading designtaxonomy
Caveats
Trajectories come from four public sets (HotpotQA, GAIA dev, BFCL, tau2), so judges may have seen the tasks; it grades judges, not agents; repo license not seen.
Details
| Workloads | tool calling, general assistant, knowledge qa |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | exact match |
| Size | 1,000 |
| Data | real public |
| Personal data | none |
| License | unknown |
| Harness | own repo |
| Status | active |