Mahal Systems SimpleQA
Fits assistants whose risk is confidently stating wrong facts. Borrow its three-way grade (correct, incorrect, not attempted) and its F-score that rewards abstaining over guessing; this is a good pattern for any agent that should say "I don't know".
What it asks the agent to do
- answer a short fact question with a single indisputable answer
- decline to answer when unsure rather than guess
What to borrow from it
grading designbaseline
Caveats
Tests memorized knowledge with no retrieval, so it says little about a RAG agent; questions have been public since 2024 and leak into training data; the simple-evals repo stopped receiving updates in July 2025. SimpleQA Verified (Google DeepMind, 2025, 1,000 prompts, arXiv 2509.07968, hosted on Kaggle) fixes label noise and topic imbalance and is the version to prefer for new runs.
Details
| Workloads | knowledge qa |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | llm judge |
| Size | 4,326 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |