Mahal Systems RoTBench
Fits teams whose tool schemas change or are poorly documented. The pattern to copy is taking a clean case set and producing slight, medium, heavy and union noise variants to measure robustness.
What it asks the agent to do
- select the right tool when tool names are garbled or swapped
- identify parameters when their names are noisy
- fill parameter values correctly under noisy documentation
What to borrow from it
case templatetaxonomy
Caveats
Only 105 base queries (840 across noise levels), first-call only, and the README still says evaluation instructions are on the way.
Details
| Workloads | tool calling, general assistant |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | 568 real tools from ToolEyes, with five noise levels applied to their docs |
| Grading | programmatic check |
| Size | 105 |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | none |
| Status | active |