Mahal Systems FunctionChat-Bench
Fits chat assistants where every turn must be one of call, answer, ask, or refuse. The four-way output taxonomy and the rubric-based pass/fail judge are a compact grading design to reuse.
What it asks the agent to do
- produce a correct tool call from a user turn
- complete the answer in natural language after a tool result
- ask a slot question when required arguments are missing
- detect that no available tool is relevant and say so
What to borrow from it
taxonomygrading design
Caveats
Korean only; the dialog set is 200 items from 45 conversations, and scoring depends on a GPT-4 judge.
Details
| Workloads | general assistant, tool calling |
|---|---|
| Interaction | multi turn scripted |
| Environment | static |
| End state checked | No |
| Tools | 25 functions across 5 tool set types, Korean dialogues |
| Grading | llm judge |
| Size | 700 |
| Data | synthetic |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |