HammerBench

Fits on-device or app assistants doing slot filling. Borrow the per-turn snapshot metrics (parameter hallucination rate, parameter missing rate, progress rate) and the four perturbation types (imperfect, Q&A trajectory, intent shift, external reference).

What it asks the agent to do

• collect missing arguments over several turns for a phone function
• handle a user who changes intent or overrides a value mid-dialogue
• resolve pronouns and references to the user's personal data
• avoid hallucinating parameter names

What to borrow from it

grading designtaxonomycase template

Caveats

Built partly from anonymized user logs with Chinese tool descriptions, dialogue is scripted, and only about 404 conversations have more than one turn.

Details

Links

• Paper
• Github