ShortcutsBench

Fits workflow-automation agents that compose many app or SaaS APIs into one long sequence. Borrow its four-way grading (API choice, parameter fill, reuse of prior outputs, asking for input) and its difficulty tiers by sequence length.

What it asks the agent to do

• choose the next app API in a long action sequence given the history so far
• fill static parameters correctly
• reuse outputs of earlier actions as inputs to later ones
• recognize when the workflow needs input from the user or the system and ask for it

What to borrow from it

case templatetaxonomygrading design

Caveats

Queries are GPT-4o-written from real shortcuts, not from users; no execution, so grading is next-step accuracy against one gold sequence; Apple-only APIs; shortcut data carries the terms of the sharing sites it was scraped from.

Details

Links

• Paper
• Github