Mahal Systems MINT
Fits agents that iterate with a user on code or quantitative answers. The useful part is the GPT-4 feedback simulator and the turn-budget analysis (k = 1 to 5 turns), which shows how much a feedback loop is worth.
What it asks the agent to do
- solve a math or reasoning problem by running Python
- write a function and fix it after execution feedback
- complete a household text-game task over several steps
- revise an answer after natural-language feedback from a simulated user
What to borrow from it
user simulatorbaseline
Caveats
Built from public datasets (GSM8K, HumanEval, MBPP, HotpotQA, ALFWorld) that are likely in training data; the repo has not been updated since 2024.
Details
| Workloads | coding, general assistant, tool calling, embodied |
|---|---|
| Interaction | multi turn simulated user |
| Environment | mixed |
| End state checked | No |
| Tools | Python code execution, ALFWorld text environment |
| Grading | exact match, unit tests |
| Size | 586 |
| Data | real public |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | frozen |