Mahal Systems Mind2Web 2
Fits research assistants whose answers depend on current web state. Borrow Agent-as-a-Judge: a per-task judge script built from a tree rubric that opens the cited pages and checks both correctness and attribution.
What it asks the agent to do
- answer a long-horizon question whose answer changes over time
- browse live websites and gather facts from many pages
- cite a URL for each key fact in the answer
What to borrow from it
grading designcase template
Caveats
130 tasks and the test set is private, so leaderboard entries go by email; judges need OpenAI access and a browser cache; building each task took many human hours, so it is hard to extend.
Details
| Workloads | web browsing, research |
|---|---|
| Interaction | autonomous long horizon |
| Environment | browser |
| End state checked | No |
| Tools | web search, page browsing |
| Grading | llm judge, programmatic check |
| Size | 130 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |