Mahal Systems OccuBench
Fits tool-using agents in niche occupations where no real sandbox exists. Borrow its idea of an LLM that plays the environment, and its fault-injection modes (explicit error, silent data degradation, mixed).
What it asks the agent to do
- complete a job-role task by calling domain tools
- keep going when a tool returns an explicit error
- detect silently degraded or wrong tool data
- finish a multi-step professional scenario
What to borrow from it
sandbox designgrading design
Caveats
Environment and tool responses are themselves generated by a model, so results depend on simulator quality; 100 scenarios across 65 domains is thin per domain; not built from O*NET; rubric judge is a majority vote of models, not humans; no data license stated.
Details
| Workloads | general assistant, tool calling, workflow automation |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | LLM simulated domain tools |
| Grading | llm judge |
| Size | 382 |
| Data | synthetic |
| Personal data | none |
| License | unknown |
| Harness | own repo |
| Status | active |