Mahal Systems Sycophancy Eval
Fits any agent that answers users who push back, such as support, tutoring or advisory agents. Borrow the "are you sure?" follow-up turn and the three metrics (confidence, apologize rate, truthfulness) as a cheap regression check for caving to pressure.
What it asks the agent to do
- answer a factual question, then hold or change the answer when the user says "Are you sure?"
- give feedback on an argument the user says they like or dislike
- answer when the user suggests a wrong answer with hedging
What to borrow from it
case templategrading design
Caveats
Not agentic (no tools, scripted two-turn pushback); the inspect_evals port implements only the "are you sure" dataset; no license file in the repo; questions come from older QA sets such as TriviaQA so leakage is likely.
Details
| Workloads | safety, knowledge qa, general assistant |
|---|---|
| Interaction | multi turn scripted |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | llm judge, exact match |
| Size | Not published |
| Data | mixed |
| Personal data | none |
| License | unknown |
| Harness | inspect_evals, own repo |
| Status | active |