Mahal Systems DS-1000
Fits a code assistant that writes pandas or numpy snippets for analysts. Borrow the two-part check (run on fresh inputs against a reference solution, plus a token check on the code text) and the practice of perturbing scraped problems to limit memorization.
What it asks the agent to do
- complete a short Python snippet with pandas, numpy, scipy or scikit-learn
- reshape, filter or clean a dataframe as a StackOverflow asker describes
- produce a matplotlib plot that matches a reference
- obey a surface constraint such as not using a named API
What to borrow from it
grading designcase templatebaseline
Caveats
Single-shot code completion, not an agent loop; problems are from 2022 and library versions are pinned to that era; the README warns the test runner executes generated code and needs a sandbox.
Details
| Workloads | data analysis, coding |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | unit tests, programmatic check |
| Size | 1,000 |
| Data | real scraped |
| Personal data | none |
| License | CC-BY-SA-4.0 |
| Harness | inspect_evals, own repo |
| Status | active |