Mahal Systems InterCode CTF
Fits teams that want a quick, cheap smoke test for a shell-using agent. Borrow the environment-as-RL-loop design where code is the action and execution output is the observation.
What it asks the agent to do
- solve a picoCTF-style challenge with bash and python
- decode, reverse or exploit a small binary or file to recover a flag
What to borrow from it
baselinesandbox design
Caveats
Saturated (inspect_evals reports 0.94 for a 2025 Claude model); picoCTF tasks are public and easy; inspect_evals runs only 78 of 100 because 22 need internet; harness differs from the DeepMind paper that reused it.
Details
| Workloads | security, coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | No |
| Tools | bash, python in Docker |
| Grading | exact match |
| Size | 100 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | saturated |