Mahal Systems ELT-Bench
Fits data engineering agents that build or maintain extract-load-transform pipelines. Borrow its two-stage grading that compares produced warehouse tables to ground-truth CSVs and its containerized source systems (835 source tables, 203 data models).
What it asks the agent to do
- set up Airbyte connectors to extract from APIs, S3, PostgreSQL, MongoDB and flat files
- load source tables into a cloud warehouse such as Snowflake
- write dbt SQL models that produce the required data models
- debug a long pipeline run until the output tables match
What to borrow from it
sandbox designgrading designcase template
Caveats
Needs a paid warehouse account and Airbyte deployment; best result in the paper was 3.9% of data models correct at about $4.30 per pipeline; the current repo (ELT-Bench++) has shifted tooling and its README no longer mentions dbt or Terraform; data provenance is not stated in the abstract or README.
Details
| Workloads | devops, sql, workflow automation, coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | Airbyte, dbt, Terraform, Snowflake or Databricks or Redshift, Docker, shell |
| Grading | state diff, programmatic check |
| Size | 100 |
| Data | mixed |
| Personal data | none |
| License | CC-BY-SA-4.0 |
| Harness | own repo |
| Status | active |