Mahal Systems SWE-PRBench
Fits code-review agents that choose how much context to load; its finding that more context lowered every model's score is directly useful for prompt design.
What it asks the agent to do
- find the issues a human reviewer flagged in a pull request diff
- review with diff only, diff plus files, or full repository context
What to borrow from it
grading designtaxonomy
Caveats
Single author, LLM-judged (kappa 0.75), and the repository URL is not on the paper page; models found only 15 to 31 percent of issues, so scores cluster near the floor.
Details
| Workloads | code review |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | llm judge |
| Size | 350 |
| Data | real public |
| Personal data | possible |
| License | CC-BY-4.0 |
| Harness | none |
| Status | active |