Mahal Systems SWE-Lancer
Fits product-engineering agents on a large JavaScript/TypeScript app, and manager-style agents that pick between proposals. Borrow dollar-weighted scoring and Playwright end-to-end tests.
What it asks the agent to do
- implement a paid freelance feature or bug fix in the Expensify React Native app
- pass hidden end-to-end browser tests written by professional engineers
- choose the best of several engineers' implementation proposals as a manager
What to borrow from it
grading designbaselinetaxonomy
Caveats
One codebase (Expensify) only; the public Diamond split is 502 tasks and the repo says only 198 of the 237 IC SWE Diamond tasks run offline; e2e tests are slow and flaky to run.
Details
| Workloads | coding, code review |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor, browser for e2e tests |
| Grading | unit tests, exact match |
| Size | 1,488 |
| Data | real public |
| Personal data | possible |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |