Mahal Systems Spider2-V
Fits agents that operate data tools through a GUI or mixed GUI plus code, such as a pipeline operator or warehouse admin assistant. Borrow its per-task evaluation functions and the VM built on OSWorld with human-validation scripts.
What it asks the agent to do
- complete a data engineering task inside a desktop app by clicking and typing
- write SQL or Python in tools such as BigQuery, dbt, Airbyte, Dagster or Snowflake
- follow tool documentation to configure a pipeline or warehouse job
- finish a task in a remote cloud workspace through its web console
What to borrow from it
sandbox designgrading designcase template
Caveats
Needs VMware and self-registered Snowflake and BigQuery accounts for part of the tasks; no public leaderboard, only 2024 baselines (GPT-4V 14.0%); heavy to set up and slow to run.
Details
| Workloads | computer use, data analysis, devops, sql |
|---|---|
| Interaction | autonomous long horizon |
| Environment | desktop |
| End state checked | Yes |
| Tools | Ubuntu VM with pyautogui mouse and keyboard, 20 data tools, browser, terminal |
| Grading | programmatic check, state diff |
| Size | 494 |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |