Mahal Systems ChemBench
Fits chemistry assistants and lab-support agents that must answer questions and know when they are unsure. Borrow its topic-level reports, numeric-tolerance parsing of answers, and the comparison against a human chemist panel.
What it asks the agent to do
- answer a chemistry question with a number or a choice
- reason about reactions, spectra and molecular properties
- answer chemical safety and toxicity questions
- say how confident the answer is
What to borrow from it
baselinetaxonomygrading design
Caveats
Question answering only, no tools or environment; top models already beat the human chemists in the paper so headline scores are near saturation; the paper notes models still fail some basic tasks and are overconfident, which the headline score hides.
Details
| Workloads | science, knowledge qa, safety |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | exact match, programmatic check |
| Size | 2,700 |
| Data | mixed |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |