Mahal Systems m&m's
Fits agents that plan a short pipeline over heterogeneous tools (OCR, captioning, weather, search) before executing. Borrow its plan-level metrics (tool-F1, argname-F1, pass rate) and its finding that step-by-step planning with feedback beats one-shot plans.
What it asks the agent to do
- plan a chain of 1-3 tools (vision models, text models, public APIs, image ops) for a request
- write the plan as JSON or as code with correct tool names and arguments
- pass one tool's output as the input of the next
- revise the plan from parsing, verification or execution feedback
What to borrow from it
grading designcase template
Caveats
Grades plans against one gold plan, so valid alternative pipelines are penalized; queries were GPT-4-written from sampled tool graphs; no LICENSE file in the repo; no updates since 2024.
Details
| Workloads | tool calling, planning, documents, general assistant |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | 33 tools (13 ML models, 9 public APIs, 11 image processing modules) |
| Grading | programmatic check, exact match |
| Size | 882 |
| Data | mixed |
| Personal data | possible |
| License | unknown |
| Harness | own repo |
| Status | frozen |