Mahal Systems NoisyToolBench
Fits any agent that gets vague requests and should ask rather than guess. Borrow the case format (imperfect query, expected clarifying question, scripted answer, expected call) and the ToolEvaluator that scores both accuracy and redundant questions.
What it asks the agent to do
- ask the right clarifying question when key information is missing
- ask when an instruction is ambiguous between tools or values
- avoid inventing arguments the user never gave
- complete the API call once the user answers
What to borrow from it
user simulatorgrading designcase template
Caveats
Only 200 cases built by corrupting ToolBench instructions, and the user answers and judging depend on GPT-4o.
Details
| Workloads | tool calling, general assistant |
|---|---|
| Interaction | multi turn simulated user |
| Environment | api sandbox |
| End state checked | No |
| Tools | 100 real RESTful APIs from ToolBench across 49 categories |
| Grading | llm judge, programmatic check |
| Size | 200 |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |
Links
Situations it helped establish
- Tool error: A tool call fails, times out or returns an error partway through.