Mahal Systems SWR-Bench
Fits pull-request review bots; borrow the structured ground-truth issue list and the coverage-based judge (about 90 percent agreement with humans) as a review rubric.
What it asks the agent to do
- review a whole pull request with full project context and point out real issues
- stay quiet on a clean pull request instead of inventing problems
What to borrow from it
grading designcase template
Caveats
Judge-based scoring, no public repository link on the paper page, and the paper title and dates differ across versions, so confirm which release you use.
Details
| Workloads | code review |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | llm judge |
| Size | 1,000 |
| Data | real public |
| Personal data | possible |
| License | unknown |
| Harness | none |
| Status | active |