Mahal Systems TelcoAgent-Bench
Fits network-operations or technical-support agents in telecom rather than consumer customer support. Borrow the metric suite (intent recognition, ordered tool execution, resolution correctness, stability across variants) and the blueprint-with-sampled-KPIs method for generating variants.
What it asks the agent to do
- recognize the network problem from an engineer's description in English or Arabic
- run the troubleshooting tools (KPI time series, neighbor audit, config push) in the required order
- ignore distractor tools that do not fit the flow
- give the correct resolution and stay consistent across rephrased variants of the same scenario
What to borrow from it
taxonomygrading design
Caveats
Agent starts from a problem statement with no interactive user; dialogues are synthetic from 49 blueprints (about 1,470 samples, exact count not clearly stated); the repo linked in the paper was not inspected, so license and runnability are unknown.
Details
| Workloads | telecom, it ops, tool calling |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | 7 telecom operations tools plus 6 distractor tools |
| Grading | trajectory check, programmatic check |
| Size | Not published |
| Data | synthetic |
| Personal data | none |
| License | unknown |
| Harness | none |
| Status | active |