Mahal Systems ToolComp
Fits research-style assistants that chain several tools to reach one answer; borrow its idea of human-corrected step labels (process supervision) next to a final-answer check, and its enterprise versus chat split.
What it asks the agent to do
- answer a multi-step question by combining search, calculator, weather, stock and Python tools
- plan and revise intermediate steps toward a verifiable final answer
- solve chat-style questions with only web search and Python
What to borrow from it
grading designtaxonomy
Caveats
The data and evaluation code are not public (release promised only when a model passes 85 percent or at the end of 2026), the final answer is graded by a GPT-4 Turbo judge, and several tools hit live web services, so you can only read the leaderboard, not run it.
Details
| Workloads | tool calling, research, finance |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | 11 tools (date, weather, calculator, Wikipedia, Google search, Wolfram Alpha, stock data, Python) |
| Grading | llm judge, trajectory check |
| Size | 485 |
| Data | mixed |
| Personal data | none |
| License | unknown |
| Harness | none |
| Status | active |