Mahal Systems IaC-Eval
Fits infrastructure-as-code agents; borrow intent specs checked with OPA against the Terraform plan instead of string-matching the HCL.
What it asks the agent to do
- write Terraform for an AWS setup described in plain language
- produce a configuration that satisfies a policy-as-code intent check
What to borrow from it
grading designcase template
Caveats
AWS and Terraform only, single-turn, and the README calls it not production-ready; GPT-4 scored 19 percent at release so it may be partly saturated by now.
Details
| Workloads | devops, it ops |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | terraform, opa |
| Grading | programmatic check |
| Size | 458 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |