Mahal Systems FailSafeQA
Fits document question answering agents in finance or any field where users upload the wrong file or type sloppy questions. Borrow its seven-condition design (clean query, three query perturbations, OCR noise, missing context, irrelevant context) and its robustness score that takes the minimum across conditions.
What it asks the agent to do
- answer a question over a long 10-K filing (4k to 27k tokens)
- handle a misspelled, incomplete or off-domain version of the question
- answer from OCR-degraded text
- refuse when the provided document is missing or irrelevant
What to borrow from it
grading designcase templateprivacy pattern
Caveats
Only 220 base items; grading relies on an LLM judge on a 1 to 6 scale; the queries are synthetic and contexts come only from SEC 10-K filings; the inspect_evals port is externally maintained at a pinned commit.
Details
| Workloads | finance, documents, safety |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | llm judge |
| Size | 220 |
| Data | mixed |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals |
| Status | active |