Mahal Systems Defects4J
Still the reference set for Java program-repair agents (RepairAgent, ReinFix report on it); borrow the minimized single-commit fix and triggering-test rules for writing repair cases.
What it asks the agent to do
- fix a real Java bug so the triggering JUnit test passes
- localize the faulty method from failing tests
What to borrow from it
case templatebaseline
Caveats
From 2014, fully in training data, and results are split across v1.2 and v2.0 subsets that papers mix up; not an agent benchmark on its own.
Details
| Workloads | coding |
|---|---|
| Interaction | single turn |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, java build |
| Grading | unit tests |
| Size | 854 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | frozen |