Mahal Systems ACEBench
Fits assistants that must refuse or ask back when a request is ambiguous or unservable, not just call tools; borrow its Normal / Special / Agent split and its rule-based scoring with end-state and process accuracy for the multi-turn subset.
What it asks the agent to do
- call the right synthetic API with correct arguments from one request
- spot a missing or wrong parameter, or a request no tool can serve, and say so
- finish a multi-turn task with a simulated user inside a sandbox
What to borrow from it
grading designtaxonomyuser simulator
Caveats
All APIs are synthetic and most cases are single-turn; the Agent subset relies on a GPT-4o user simulator; the per-category counts are not given on the README, and the English and Chinese halves are parallel translations of the same 2,000 entries.
Details
| Workloads | tool calling |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | Yes |
| Tools | 4,538 synthetic APIs across 8 domains and 68 sub domains, sandbox environments for the Agent subset |
| Grading | programmatic check, state diff, trajectory check |
| Size | 2,000 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |