Mahal Systems UltraTool
Fits agents that plan multi-step work over a large tool catalog, and teams that want to grade planning separately from execution. Borrow its planning/creation/usage split, its six-dimension GPT-4 plan rubric, and its key-value accuracy for tool selection and arguments.
What it asks the agent to do
- write a natural-language step plan for a complex request before any tool is chosen
- decide whether the available tools are enough or a new tool must be defined
- define a missing tool (name, description, parameters)
- select the right tool for a step and fill its arguments, including nested calls
What to borrow from it
taxonomycase templategrading design
Caveats
No tools are executed, so nothing checks that a plan would actually work; planning and tool-creation scores depend on GPT-4 as judge; originally Chinese with GPT-4 translation to English; 1,000-sample test split out of 5,824 total.
Details
| Workloads | tool calling, planning, general assistant |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | 2,032 tool definitions across 22 domains (no execution) |
| Grading | llm judge, exact match, programmatic check |
| Size | 1,000 |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | frozen |