Mahal Systems DataClawEval
Fits data engineering agents that write jobs for several engines and must verify their own output. Borrow its deterministic rule-based graders and its score that blends artifact quality with process quality (exploration, efficiency, self-verification).
What it asks the agent to do
- write a batch job in PySpark, HiveSQL, MySQL or Trino from a business request
- write a streaming FlinkSQL job
- explore the schema and verify the output before finishing
- produce output tables with the right schema, rows and numbers
What to borrow from it
grading designsandbox design
Caveats
Very new (July 2026) with scores already around 75 so headroom is limited; half the prompts are in Chinese; the harness is wired to specific coding-agent backends (Claude Code, Codex, CodeBuddy); process points reward behavior not just results.
Details
| Workloads | devops, sql, coding, data analysis |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | PySpark, HiveSQL, MySQL, Trino, FlinkSQL in a fresh Docker container, shell |
| Grading | programmatic check, state diff, trajectory check |
| Size | 100 |
| Data | mixed |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |