Mahal Systems SciKnowEval
Fits science question-answering assistants in biology, chemistry, physics or materials. Borrow the five-level taxonomy (memory, comprehension, reasoning, discernment, application) as a way to bucket an agent's own test cases by depth.
What it asks the agent to do
- recall a scientific fact
- explain or comprehend a scientific text
- reason through a multi-step science problem
- judge whether a request is safe or ethical
- apply knowledge to a practical task such as property prediction
What to borrow from it
taxonomybaseline
Caveats
Question answering only, not agentic; no license stated in the repository; some tasks need GPT-4o as grader and are skipped without a key; relation-extraction tasks are scored by word-vector similarity, which is a loose check.
Details
| Workloads | science, knowledge qa, safety |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | exact match, llm judge, programmatic check |
| Size | 28,392 |
| Data | mixed |
| Personal data | none |
| License | unknown |
| Harness | inspect_evals, own repo |
| Status | active |