Mahal Systems Ask or Assume? (clarification-seeking in coding agents)
Fits coding agents and any agent that should ask before acting on vague requests. Borrow the recipe of stripping details from SWE-bench Verified issues and scoring both the resolve rate and how many questions were asked.
What it asks the agent to do
- notice that a software issue description is underspecified
- ask a clarifying question instead of guessing
- resolve the issue once the missing detail is supplied
What to borrow from it
case templateuser simulator
Caveats
Not a named benchmark release; the paper evaluates its own scaffold on an underspecified SWE-bench variant, task count and code link are not on the paper page, paper is CC BY-NC-SA.
Details
| Workloads | coding, general assistant |
|---|---|
| Interaction | multi turn simulated user |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, code editing, test runner |
| Grading | unit tests |
| Size | Not published |
| Data | real public |
| Personal data | none |
| License | unknown |
| Harness | none |
| Status | active |
Links
Situations it helped establish
- Ambiguous: The request matches more than one thing, or is too vague to act on without a question or a look at the data.