Mahal Systems PatchEval
Fits vulnerability-repair agents in web languages; borrow the paired security plus functionality test as the pass rule.
What it asks the agent to do
- patch a real CVE in a Go, JavaScript or Python project
- pass both the security test and the existing functionality tests in a sandbox
What to borrow from it
grading designcase template
Caveats
Only 230 of the 1,000 CVEs have runtime sandboxes, so most are static; code link and data license were not on the paper page.
Details
| Workloads | security, coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor |
| Grading | unit tests, programmatic check |
| Size | 1,000 |
| Data | real public |
| Personal data | none |
| License | unknown |
| Harness | inspect_evals |
| Status | active |