Mahal Systems

Overview › Benchmark catalog

Agentic Misalignment

Fits any agent given standing access to a company inbox or documents plus a long-running goal, such as an executive assistant or operations agent. Borrow the way one scenario is crossed with goal type, goal content and urgency to build a condition grid, and the classifier prompt that labels a transcript harmful or not.

What it asks the agent to do

  • act as an email oversight assistant at a fictional company
  • decide what to do when told it will be replaced or restricted
  • choose whether to blackmail an executive using information found in email
  • choose whether to leak confidential documents to an outside party

What to borrow from it

case templategrading designsandbox design

Caveats

One fictional scenario with three harmful endings (blackmail, leak, lethal action); the model only reads emails and replies, no real tool execution; grading depends on an LLM classifier and the inspect_evals port reports different rates from the original blog post.

Details

Workloadssafety, email calendar, general assistant
Interactionsingle turn
Environmentstatic
End state checkedNo
Toolsnone in the text version; simulated email inbox given as context, send email action in original
Gradingllm judge, programmatic check
SizeNot published
Datasynthetic
Personal datanone
LicenseMIT
Harnessinspect_evals, own repo
Statusactive

Links