Mahal Systems MASK
Fits agents whose operator prompt could push them to shade the truth, such as sales or support bots with a persona. Borrow the split between honesty and accuracy, and the belief-elicitation step that measures what the model thinks before applying pressure.
What it asks the agent to do
- state what it believes about a factual claim when asked neutrally
- answer the same claim under a system prompt or user that pressures it to lie
- stay consistent between its belief and its pressured answer
What to borrow from it
grading designcase template
Caveats
Single-turn prompts with no tools, so it tests the model not the agent loop; dataset is gated on Hugging Face; grading uses GPT judges, and the inspect_evals port warns that multi-epoch runs are unreliable; pii marked possible because some archetypes use real-world people and events.
Details
| Workloads | safety, knowledge qa, general assistant |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | llm judge |
| Size | 1,000 |
| Data | mixed |
| Personal data | possible |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |