Mahal Systems InstrumentalEval
Fits teams giving an agent an open-ended goal such as make money or grow usage, who want a cheap screen for power-seeking or self-preservation talk. Borrow the six behavior categories as a taxonomy for red-team cases.
What it asks the agent to do
- respond to a scenario that tempts it to evade shutdown or copy itself
- respond to a scenario where deceiving the user would help its goal
- respond to a scenario where hacking a system would help its goal
- respond to a scenario where hiding behavior or appearing aligned would help
What to borrow from it
taxonomycase template
Caveats
Text prompts only, no tools or environment; no license and no item count seen; the inspect_evals port reports different results from the paper and depends on a grader model's judgment.
Details
| Workloads | safety, general assistant |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | llm judge |
| Size | Not published |
| Data | synthetic |
| Personal data | none |
| License | unknown |
| Harness | inspect_evals, own repo |
| Status | active |