Mahal Systems T-Eval
Fits teams that want to diagnose which step of tool use their agent gets wrong (format, tool choice, plan, reasoning, argument extraction, response review) rather than only the end result. Borrow its decomposition and its per-step graders.
What it asks the agent to do
- produce a tool call in the required format with correct parameters
- pick the right tool by name from a list
- write a multi-step plan before calling tools
- state the reasoning for the next step
- read a tool's response and extract the needed arguments
- judge whether a tool response is correct, incomplete or an error
What to borrow from it
taxonomygrading designcase template
Caveats
Queries and gold solutions are GPT-3.5/GPT-4 generated over only 553 base queries; the Plan/Reason/Understand splits use sentence-embedding similarity, which is a soft signal; no updates since 2024 and the leaderboard is stale.
Details
| Workloads | tool calling, planning |
|---|---|
| Interaction | multi turn scripted |
| Environment | static |
| End state checked | No |
| Tools | 15 tools across research, travel, entertainment, web, life, finance |
| Grading | exact match, programmatic check, llm judge |
| Size | 23,305 |
| Data | synthetic |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | frozen |