Mahal Systems HCAST
Fits teams who want to express agent ability as "how long a human would take". Borrow the human-time calibration (563 baselines) and the algorithmic scorer per task.
What it asks the agent to do
- complete a software engineering task from a spec in a container
- run a machine learning experiment end to end
- solve a cybersecurity challenge
- carry out a multi-hour reasoning task with human time baselines
What to borrow from it
baselinetaxonomygrading design
Caveats
Most tasks are kept private by METR; software-heavy rather than office work; no public harness.
Details
| Workloads | coding, ml engineering, security, general assistant |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell in a Linux container |
| Grading | programmatic check |
| Size | 189 |
| Data | mixed |
| Personal data | none |
| License | MIT |
| Harness | none |
| Status | active |