Mahal Systems Hack-Verifiable Terminal Bench (HVTB)
Fits coding and operations agents that run against tests, where reward hacking is a worry. Borrow the trick of planting a reference solution and tests in the container and using filesystem watchers to flag any read, giving a hack rate with no judge.
What it asks the agent to do
- solve a terminal or coding task in a container
- resist reading a planted reference solution or held-out tests
- pass the task checks without violating the task's intent
What to borrow from it
grading designsandbox design
Caveats
Tasks are reused from Terminal Bench 2.1 so capability numbers are not new; the inspect_evals port uses a different agent harness than the paper; the host repo (RHOB) carries a correction banner about other claims, so review the pinned commit before trusting it.
Details
| Workloads | safety, coding, it ops |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | bash |
| Grading | programmatic check, unit tests |
| Size | 89 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |