Mahal Systems LongBench v2
Fits any agent that must read long inputs (8k to 2M words) before answering, such as contract review or codebase question answering. Borrow its six-category task split and its human-written, time-limited question style, which keeps questions hard to shortcut.
What it asks the agent to do
- answer a multiple-choice question about one long document
- answer a question that needs facts from several long documents
- recall a detail from a very long dialogue history
- answer a question about a whole code repository
- read a long structured file such as a table or log
What to borrow from it
baselinecase templatetaxonomy
Caveats
Multiple choice only, so it does not test free-form answers or extraction; 503 questions across six categories leaves each category small; human experts score only 53.7% under a 15-minute limit, so the ceiling is unclear.
Details
| Workloads | documents, knowledge qa, coding, research |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | exact match |
| Size | 503 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |