Mahal Systems Aider polyglot benchmark
Fits chat-style code editing tools that apply diffs to files; borrow the two-attempt protocol (pass rate 1 vs 2) that measures recovery from test feedback.
What it asks the agent to do
- solve an Exercism exercise in C++, Go, Java, JavaScript, Python or Rust by editing files
- fix the solution after seeing test failures on a second attempt
What to borrow from it
baselinegrading design
Caveats
Top models are near 88 percent so headroom is small; exercises are public Exercism problems (likely memorized) and the content license belongs to Exercism.
Details
| Workloads | coding |
|---|---|
| Interaction | multi turn scripted |
| Environment | containers |
| End state checked | Yes |
| Tools | file editor, test runner |
| Grading | unit tests |
| Size | 225 |
| Data | real public |
| Personal data | none |
| License | unknown |
| Harness | own repo |
| Status | active |