Mahal Systems SycoBench-600
Fits agents that must hold a correct answer under user pressure but still take valid corrections, such as support or advisory bots. Borrow the correction-selectivity metric (update rate minus wrong-flip rate) and the matched correct-suggestion turn.
What it asks the agent to do
- answer a multiple-choice question
- keep the answer under three turns of doubt, false authority or a wrong suggestion
- accept a correct suggestion when the first answer was wrong
What to borrow from it
grading designcase template
Caveats
Multiple-choice only with no tools, so it measures the model not the agent; single author; headline numbers use a 555-question subset; dataset is CC-BY-4.0 while code is MIT.
Details
| Workloads | safety, knowledge qa |
|---|---|
| Interaction | multi turn scripted |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | exact match |
| Size | 600 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |