Mahal Systems CRMArena
Fits agents that read a real CRM (Salesforce or similar) to answer service, analyst or manager questions. Borrow the persona-based task taxonomy, the 30% unanswerable queries, and the way a synthetic org is generated from a schema with hidden latent variables that drive the ground truth.
What it asks the agent to do
- route a service case to the right agent given skills and workload
- find the policy violation or the right knowledge article for a case
- identify a named-entity or transfer-count metric from the CRM records
- answer an analyst or manager question by querying linked CRM objects
- say "None" when the question has no answer in the data
What to borrow from it
sandbox designprivacy patterntaxonomybaseline
Caveats
Single-turn and read-only (no writes to the CRM, no customer in the loop); non-commercial license; nine task types with 130 instances each; requires access to Salesforce orgs that the authors host, and the GUI access is no longer public.
Details
| Workloads | crm, customer support, knowledge qa, tool calling |
|---|---|
| Interaction | single turn |
| Environment | database |
| End state checked | No |
| Tools | live Salesforce org through the Simple Salesforce API, SOQL and SOSL queries, record lookup tools |
| Grading | exact match, programmatic check |
| Size | 1,170 |
| Data | synthetic |
| Personal data | none |
| License | CC-BY-NC-4.0 |
| Harness | own repo |
| Status | active |
Links
Situations it helped establish
- Unanswerable: The data holds no answer, or the tools can't do what was asked.
- Different asker: The same job asked by a different role, such as a manager wanting an aggregate where a rep wants one record.