Mahal Systems SciCode
Fits scientific-coding assistants; borrow the main-problem versus subproblem scoring so partial progress is visible.
What it asks the agent to do
- implement a research computation in Python from a scientist-written description
- solve a chain of subproblems that build on each other in one scientific problem
What to borrow from it
case templatebaseline
Caveats
Only 80 main problems (338 subproblems); best model solved 4.6 percent at release, so early numbers are mostly zero; no tool use or repository context.
Details
| Workloads | science, coding |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | unit tests |
| Size | 80 |
| Data | synthetic |
| Personal data | none |
| License | Apache-2.0 |
| Harness | inspect_evals, hal, own repo |
| Status | active |