Mahal Systems ComplexFuncBench
Fits travel or booking agents and any agent with long, constrained parameter lists; borrow ComplexEval's layered matching (format check, exact match, then equal-response match, then a GPT-4o tie-break) for grading calls that have more than one right form.
What it asks the agent to do
- plan a multi-step booking query (hotel, flight, attraction, car rental, taxi) with dependent calls
- respect user constraints when choosing parameter values
- infer parameter values implied rather than stated
- fill long parameter values and work with up to 128k tokens of context
What to borrow from it
grading designcase template
Caveats
Single vertical (Booking.com), response matching needs a paid RapidAPI subscription, the final-response score is a GPT-4o rating, and the repository states no license (it has moved from THUDM to zai-org).
Details
| Workloads | tool calling, travel |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | 43 Booking.com APIs (via RapidAPI) with recorded golden responses |
| Grading | programmatic check, llm judge |
| Size | 1,000 |
| Data | mixed |
| Personal data | none |
| License | unknown |
| Harness | own repo |
| Status | active |