Mahal Systems ExCyTIn-Bench
Fits security operations or log-investigation agents that query a SIEM. Borrow the way questions are generated from an investigation graph so each has an explainable answer, and the controlled tenant that yields realistic logs without customer data.
What it asks the agent to do
- investigate a simulated attack from Microsoft Sentinel logs
- write SQL queries over 57 log tables to follow a chain of evidence
- answer a question about the incident such as which host or account was involved
What to borrow from it
case templategrading designprivacy pattern
Caveats
Question counts differ between sources (the arXiv abstract says 7,542 generated questions, other listings cite 589 evaluation questions) so no size is given; eight incidents only; needs Docker MySQL and GPT-4o for grading.
Details
| Workloads | security, sql, it ops |
|---|---|
| Interaction | autonomous long horizon |
| Environment | database |
| End state checked | No |
| Tools | SQL query tool over a MySQL copy of Azure security logs |
| Grading | llm judge, exact match |
| Size | Not published |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |