OmniACT

Fits teams training grounding-plus-planning in one shot. Borrow the sequence score and action score (click-in-box, key set, typed-text BLEU).

What it asks the agent to do

• write a PyAutoGUI script that performs a task from one screenshot
• locate the right element to click, type or hotkey on macOS, Linux, Windows or web apps

What to borrow from it

grading designtaxonomy

Caveats

No execution, scripts are compared to a gold script; one screenshot per task so no multi-screen tasks; repo location and license not confirmed.

Details

Links

• Paper