Mahal Systems BrowseComp
Fits agents whose job is to dig up obscure facts on the live web and return a short answer. Borrow the pattern of building questions backward from a short verifiable answer, and the accuracy plus calibration reporting.
What it asks the agent to do
- find a hard-to-locate fact by chaining many web searches
- identify a person, work or event from several intertwined clues
- give a short answer that can be checked against a reference
What to borrow from it
case templatebaselinegrading design
Caveats
Live web results drift over time, so scores are not reproducible; short answers only, no report writing; the simple-evals repo is no longer updated for new models.
Details
| Workloads | web browsing, research, knowledge qa |
|---|---|
| Interaction | autonomous long horizon |
| Environment | browser |
| End state checked | No |
| Tools | web search, page browsing |
| Grading | llm judge |
| Size | 1,266 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |