Mahal Systems Multi-IaC-Bench
Fits agents that edit infrastructure templates rather than write them fresh; the (initial template, change request, target template) triplet is the case format to copy.
What it asks the agent to do
- modify an existing CloudFormation, Terraform or CDK template per a change request
- keep the template syntactically valid and semantically aligned with the request
What to borrow from it
case template
Caveats
Synthetic, no public code link on the paper page, task count unstated, and models are already above 95 percent on syntax so only semantic checks separate them.
Details
| Workloads | devops, it ops |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | iac validators |
| Grading | programmatic check, llm judge |
| Size | Not published |
| Data | synthetic |
| Personal data | none |
| License | unknown |
| Harness | none |
| Status | active |