Mahal Systems LiveCodeBench
A model-level coding baseline rather than an agent test; borrow the date-windowed release scheme that lets you report only problems newer than a model's cutoff.
What it asks the agent to do
- solve a LeetCode, AtCoder or Codeforces problem in Python
- repair a wrong solution given test feedback
- predict the output of code for a given input
What to borrow from it
privacy patternbaseline
Caveats
Competitive programming only, no repository context or tools; scores vary with the date window chosen, so always state the release version (v6 has 1,055 problems).
Details
| Workloads | coding |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | unit tests |
| Size | 1,055 |
| Data | real scraped |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |