Mahal Systems RepoBench
Fits autocomplete or inline-suggestion products, not agents; borrow exact match plus edit similarity plus CodeBLEU as cheap text-level grading for completion cases.
What it asks the agent to do
- predict the next line of code using context from other files in the repo
- retrieve the cross-file snippet needed to complete a line
What to borrow from it
grading design
Caveats
Single-line completion only, Python and Java, no execution; data is from 2023 GitHub and likely in training sets.
Details
| Workloads | coding |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | exact match, programmatic check |
| Size | Not published |
| Data | real public |
| Personal data | none |
| License | CC-BY-4.0 |
| Harness | own repo |
| Status | frozen |