Mahal Systems CRMArena-Pro
Fits sales-assist and service agents that work inside a CRM, in both single-shot and conversational modes. Borrow the 19-task skill taxonomy (workflow execution, policy compliance, information retrieval, textual reasoning), the paired single-turn versus multi-turn scoring, and the confidentiality-awareness probes.
What it asks the agent to do
- resolve a service, sales or configure-price-quote request against B2B or B2C CRM records
- clarify a vague request over several turns with a persona-driven user, then answer
- execute a defined CRM workflow such as case routing or lead conversion
- refuse to disclose confidential records when asked
- answer a knowledge question from the org's articles
What to borrow from it
taxonomysandbox designprivacy patternbaseline
Caveats
Non-commercial license; the user simulator and answer extraction are both GPT-4o so results depend on that model; grading is answer matching rather than end-state checks; data is LLM-generated and only about two thirds of expert raters called it realistic.
Details
| Workloads | crm, sales, customer support, knowledge qa, tool calling |
|---|---|
| Interaction | multi turn simulated user |
| Environment | database |
| End state checked | No |
| Tools | live Salesforce B2B and B2C orgs through the Simple Salesforce API, SOQL and SOSL queries |
| Grading | exact match, programmatic check, llm judge |
| Size | 4,280 |
| Data | synthetic |
| Personal data | none |
| License | CC-BY-NC-4.0 |
| Harness | own repo |
| Status | active |
Links
Situations it helped establish
- Ambiguous: The request matches more than one thing, or is too vague to act on without a question or a look at the data.
- Forbidden or confirm-first: The policy forbids the request, or it needs the person's confirmation before the agent acts.
- Confidential: The request asks for data the asker may not see, such as another person's private details or a record outside their role.
- Multi-turn pair: The same request arrives over several turns, with details given late or changed, and is scored against its single-turn twin.
- Different asker: The same job asked by a different role, such as a manager wanting an aggregate where a rep wants one record.