Mahal Systems AgentRx
Fits teams debugging a deployed agent across API workflows, incident management or web tasks. Borrow the constraint-synthesis-then-check pattern that gives an auditable violation log before the judge decides.
What it asks the agent to do
- locate the critical failing step in a failed agent run
- assign the failure to a category from a cross-domain taxonomy
- check a trajectory against synthesized constraints and log violations
What to borrow from it
taxonomygrading design
Caveats
Only 170 failed runs over 11 settings; the dataset on Hugging Face is gated; no public code link was seen on the paper page.
Details
| Workloads | general assistant, it ops, tool calling, web browsing |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | exact match, llm judge |
| Size | 170 |
| Data | real public |
| Personal data | none |
| License | CC-BY-4.0 |
| Harness | none |
| Status | active |