Mahal Systems TeleCom-Bench
Fits network-operations agents that triage faults and run optimization workflows. Borrow its six-step decomposition (intent, entities, tool call, event check, root cause, solution) and its mix of macro-F1, exact match and a 5-point judge with reported rater agreement.
What it asks the agent to do
- answer telecom fundamentals and 3GPP protocol questions
- recognize operator intent and extract entities from a ticket
- pick the tool to call for a network-optimization step
- diagnose the root cause of a network fault
- propose a fix for a network fault
What to borrow from it
taxonomygrading designbaseline
Caveats
Only a subset of the 22,678 samples is public (full set on request to prevent leakage); data license is CDLA-Permissive-2.0 while code is Apache-2.0; tasks are graded as static steps from recorded trajectories, not live agent runs; vendor-specific product knowledge.
Details
| Workloads | telecom, it ops, tool calling, knowledge qa |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | tool call selection graded against trajectories |
| Grading | exact match, programmatic check, llm judge |
| Size | 22,678 |
| Data | real public |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |