Mahal Systems ITBench
The closest public match for SRE or on-call agents; borrow the push-button Kubernetes incident scenarios and the resolve-rate reporting by domain.
What it asks the agent to do
- diagnose and fix a live Kubernetes incident from alerts, logs, traces and metrics
- bring a cluster into compliance with a security policy
- cut cloud cost in a FinOps scenario
What to borrow from it
sandbox designcase templatebaseline
Caveats
Only a small subset of scenarios is open source (6 SRE, 4 CISO categories, 1 FinOps); live clusters are slow and non-deterministic; the paper license is CC BY-NC-ND.
Details
| Workloads | it ops, devops, security |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | kubectl, observability APIs, shell |
| Grading | state diff, programmatic check |
| Size | 94 |
| Data | synthetic |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |