Mahal Systems ToolTalk
Fits personal-assistant agents that act on calendars, email and messages in a conversation. Borrow the simulated tool implementations with mock databases and the strict success rule (perfect recall of required calls, zero wrong actions).
What it asks the agent to do
- set an alarm or reminder from a chat request
- send an email or message on the user's behalf
- add or change a calendar event over several turns
- look up the weather or an account detail
- carry a multi-step request across turns without redundant actions
What to borrow from it
case templategrading designsandbox design
Caveats
Only 78 handwritten conversations, the user side is scripted, and the repo has not moved since 2024; results from 2023 models are saturated by current ones.
Details
| Workloads | general assistant, email calendar, tool calling |
|---|---|
| Interaction | multi turn scripted |
| Environment | api sandbox |
| End state checked | No |
| Tools | 28 simulated personal assistant tools in 7 plugins (account, alarm, calendar, email, message, reminder, weather) |
| Grading | exact match, trajectory check |
| Size | 78 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | frozen |