SOPBench (customer service)

Fits support agents whose work is defined by written procedures with preconditions (check this, then do that). Borrow the idea of encoding each procedure as code and using it as the oracle verifier for outcome, step and trajectory compliance, so no LLM judge is needed.

What it asks the agent to do

• verify the user's identity and eligibility before taking a service action
• follow a multi-step standard operating procedure for a bank, hotel, library, DMV, clinic, market or university request
• call the right tools in the order the procedure requires
• refuse an action whose constraints are not met, including under jailbreak pressure

What to borrow from it

grading designcase templatetaxonomy

Caveats

Task count differs between README (830) and abstract (900+); data license is CC-BY-4.0 while code is MIT; tasks and databases are LLM-generated; the user simulator is off by default so many reported runs are effectively single-user-turn.

Details

Links

• Paper
• Github