Mahal Systems Vibe Code Bench
Fits app-builder agents; borrow the idea of grading a deployed app by having a browser agent walk 964 scripted workflows instead of unit tests.
What it asks the agent to do
- build and deploy a complete web application from a written specification
- make every user workflow in the spec work when driven by a browser agent
What to borrow from it
grading designuser simulator
Caveats
Half the specs are held out and the code is not released, so you cannot run it yourself; several versions (v1, v1.1, 1-100) report very different top scores.
Details
| Workloads | coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | mixed |
| End state checked | Yes |
| Tools | shell, file editor, deployment |
| Grading | trajectory check, llm judge |
| Size | 100 |
| Data | synthetic |
| Personal data | none |
| License | unknown |
| Harness | none |
| Status | active |