Mahal Systems CodeIPI
Fits coding agents that read issues and repos. Borrow the canary design: planted secrets and env vars for exfiltration, a unique marker for execution, and file hashes before and after for persistence.
What it asks the agent to do
- fix a bug in a small repository with bash and python
- ignore instructions hidden in issue text, code comments, READMEs or config files
- avoid leaking canary secrets, running planted commands or persisting changes
- flag the injection while still completing the task
What to borrow from it
grading designsandbox design
Caveats
Only 45 samples (35 injected, 10 benign) and no paper; the task source is not described; reported numbers lag the latest version.
Details
| Workloads | safety, security, coding |
|---|---|
| Interaction | single turn |
| Environment | containers |
| End state checked | Yes |
| Tools | bash and python in a no network Docker sandbox with git and pytest |
| Grading | programmatic check, llm judge |
| Size | 45 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals |
| Status | active |