Mahal Systems Spider 2.0
Fits analytics or warehouse agents that must answer questions over big real schemas in cloud SQL dialects. Borrow its three sub-variants: Spider2-Lite (547 tasks, BigQuery 214, Snowflake 198, SQLite 135), Spider2-Snow (547 tasks on Snowflake only) and Spider2-DBT (68 repository-level dbt tasks on DuckDB), and its execution-result grading.
What it asks the agent to do
- answer a business question by writing one or more SQL queries over a large warehouse
- find the right tables and columns in metadata with more than 1,000 columns
- read dialect documentation and write BigQuery, Snowflake or SQLite SQL
- write or fix dbt transformation models in a project repository (Spider2-DBT)
- run multi-step SQL workflows that chain queries and transform data
What to borrow from it
baselinecase templatesandbox design
Caveats
Only partial gold answers are public and the rest go through leaderboard submission; BigQuery tasks cost money; an August 2026 repo notice reports Spider2-Snow access disrupted by a Snowflake account suspension; best reported score at release was 21.3%.
Details
| Workloads | sql, data analysis, coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | database |
| End state checked | Yes |
| Tools | SQL execution on BigQuery, Snowflake, SQLite or DuckDB, file reads in a dbt project |
| Grading | programmatic check, state diff |
| Size | 632 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |