Mahal Systems SealQA
Fits search-augmented assistants that must not be fooled by bad search hits. Borrow the idea of selecting questions where search returns conflicting or unhelpful results, and the LongSeal needle-in-a-haystack variant for retrieval-heavy agents.
What it asks the agent to do
- answer a fact question when search results conflict or are noisy
- decide which of several sources to trust
- find the relevant document among many distractors in long context
What to borrow from it
case templatebaseline
Caveats
Seal-0 is only 111 questions (Seal-Hard and LongSeal have 254 each); grading is a Colab template rather than a runnable repo; live search so difficulty shifts over time.
Details
| Workloads | web browsing, knowledge qa |
|---|---|
| Interaction | single turn |
| Environment | browser |
| End state checked | No |
| Tools | web search |
| Grading | llm judge |
| Size | 111 |
| Data | real public |
| Personal data | none |
| License | Apache-2.0 |
| Harness | none |
| Status | active |