Mahal Systems AgentBench
Fits teams who want one harness covering shell, database and web-style tasks to sanity-check a general agent. Borrow the containerized per-environment servers; the 2025 function-calling version keeps it usable with current tool-call APIs.
What it asks the agent to do
- run shell commands to complete an operating system task
- write and run SQL to answer or change a database
- query a knowledge graph with tools
- play a digital card game against an opponent
- complete household tasks in a text world
- shop or browse on simulated web pages
What to borrow from it
sandbox designbaselinetaxonomy
Caveats
Several sub-environments wrap older benchmarks (ALFWorld, WebShop, Mind2Web) that are near ceiling for recent models; per-environment task counts are only in an image in the README.
Details
| Workloads | general assistant, it ops, sql, shopping, games |
|---|---|
| Interaction | multi turn scripted |
| Environment | mixed |
| End state checked | Yes |
| Tools | shell, SQL, knowledge graph API, game and web simulators |
| Grading | programmatic check, state diff |
| Size | Not published |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | inspect_evals, own repo |
| Status | active |