Mahal Systems FrontierCS
Fits agents that optimize rather than just pass; borrow the continuous scoring curve (60 points for matching the reference, log bonus for beating it, capped at 100).
What it asks the agent to do
- write a program that scores as well as possible on an open-ended algorithmic optimization problem
- solve a research-style systems, HPC, database or security task with no known optimal answer
What to borrow from it
grading design
Caveats
Size is 180 algorithmic plus 68 research problems and keeps growing, so pin a version; submissions go by email, so the leaderboard is not independently run.
Details
| Workloads | coding, research |
|---|---|
| Interaction | single turn |
| Environment | containers |
| End state checked | No |
| Tools | none |
| Grading | programmatic check |
| Size | 248 |
| Data | mixed |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |