Mahal Systems CTI-REALM
Fits detection-engineering or security-operations agents. Borrow the weighted checkpoint scoring that credits CTI analysis, MITRE mapping, data exploration and query execution on the way to the final rule, and the F1 scoring of a detection rule against ground truth.
What it asks the agent to do
- read a cyber threat intelligence report
- map the described behavior to MITRE ATT&CK techniques
- explore log schemas and run KQL queries against an emulated attack
- write a KQL or Sigma detection rule that catches the attack
What to borrow from it
grading designcase template
Caveats
Only 50 tasks; the CTI-analysis judge correlates weakly with human labels (r=0.48); needs a 4.5 GB Docker image with a Kusto emulator; no dataset license stated; no standalone repo seen outside inspect_evals and a Hugging Face dataset.
Details
| Workloads | security, it ops, tool calling |
|---|---|
| Interaction | autonomous long horizon |
| Environment | mixed |
| End state checked | No |
| Tools | CTI report retrieval, KQL execution against a Kusto emulator, Sigma rule retrieval, MITRE mapping, bash, python |
| Grading | trajectory check, llm judge, programmatic check |
| Size | 50 |
| Data | synthetic |
| Personal data | none |
| License | unknown |
| Harness | inspect_evals |
| Status | active |