Mahal Systems Agents4PLC benchmark
Fits PLC or industrial-control code-generation agents. Borrow its three-stage gate (compiles, verifiable, passes formal properties) and its task format of requirement plus human-verified formal spec plus reference Structured Text.
What it asks the agent to do
- write IEC 61131-3 Structured Text from a natural-language requirement
- make the code compile
- satisfy formal properties checked by a model checker
- fix code after a failed verification
What to borrow from it
grading designcase template
Caveats
Small (23 v1 tasks, 73 v2 tasks, 21 industrial tasks); the full multi-agent framework is not released because of industrial partner agreements, only benchmark data, verifiers and an LLM4PLC baseline; Structured Text only, no ladder logic or vendor dialects.
Details
| Workloads | manufacturing, coding |
|---|---|
| Interaction | single turn |
| Environment | terminal |
| End state checked | No |
| Tools | Structured Text compiler, nuXmv model checker, PLCverif property patterns |
| Grading | programmatic check |
| Size | 96 |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |