Mahal Systems ContextCRBench
Fits teams building a review bot with issue context; borrow the three granularities (hunk, line, comment) as a taxonomy and the context-enrichment fields per record.
What it asks the agent to do
- judge whether a diff hunk needs review comments
- point to the exact lines with defects
- write the review comment a human would leave, given the linked issue and surrounding code
What to borrow from it
taxonomycase template
Caveats
Scraped from 100 public repos with no stated license and reviewer names possibly included; metrics are not described in the README; very large so sample it.
Details
| Workloads | code review |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | programmatic check, llm judge |
| Size | 67,910 |
| Data | real scraped |
| Personal data | possible |
| License | unknown |
| Harness | own repo |
| Status | active |