Mahal Systems Lost in Conversation
Fits any chat agent whose users reveal requirements gradually. Borrow the sharded instruction format, the simulator that reveals one shard per turn, and the split of results into aptitude (90th percentile) and unreliability (90th minus 10th).
What it asks the agent to do
- complete a task whose requirements arrive one piece per turn
- write code or SQL after several underspecified turns
- produce an API call once enough details are revealed
- avoid committing to an early answer and refusing to revise it
What to borrow from it
user simulatorgrading design
Caveats
Not a leaderboard benchmark but a simulation study; the 600 instructions are derived from existing public datasets and the simulated user is GPT-4o-mini.
Details
| Workloads | general assistant, coding, sql, tool calling, documents |
|---|---|
| Interaction | multi turn simulated user |
| Environment | static |
| End state checked | No |
| Tools | none; tasks graded by task specific tests or references |
| Grading | unit tests, exact match, llm judge |
| Size | 600 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |