Mahal Systems AssistantBench
Fits research assistants that browse the live web for a user. Borrow the answer-rate versus accuracy split so that abstaining is counted differently from being wrong.
What it asks the agent to do
- find a specific answer that needs browsing several live websites
- compare prices, listings or businesses across sites
- compute a figure from data scattered on the open web
What to borrow from it
grading designbaseline
Caveats
Live websites drift so answers go stale; read-only; only 33 dev tasks have public answers.
Details
| Workloads | web browsing, research, general assistant, shopping |
|---|---|
| Interaction | single turn |
| Environment | browser |
| End state checked | No |
| Tools | web browser, search |
| Grading | exact match, programmatic check |
| Size | 214 |
| Data | real public |
| Personal data | none |
| License | Apache-2.0 |
| Harness | inspect_evals, hal, own repo |
| Status | active |