Graphical User Interface (GUI) automation holds significant promise for
enhancing human productivity by assisting with computer tasks. Existing task
formulations primarily focus on simple tasks that can be specified by a single,
language-only instruction, such as "Insert a new slide." In this work, we
introduce VideoGUI, a novel multi-modal benchmark designed to evaluate GUI
assistants on visual-centric GUI tasks. Sourced from high-quality web
instructional videos, our benchmark focuses on tasks involving professional and
novel software (e.g., Adobe Photoshop or Stable Diffusion WebUI) and complex
activities (e.g., video editing). VideoGUI evaluates GUI assistants through a
hierarchical process, allowing for identification of the specific levels at
which they may fail: (i) high-level planning: reconstruct procedural subtasks
from visual conditions without language descriptions; (ii) middle-level
planning: generate sequences of precise action narrations based on visual state
(i.e., screenshot) and goals; (iii) atomic action execution: perform specific
actions such as accurately clicking designated elements. For each level, we
design evaluation metrics across individual dimensions to provide clear
signals, such as individual performance in clicking, dragging, typing, and
scrolling for atomic action execution. Our evaluation on VideoGUI reveals that
even the SoTA large multimodal model GPT4o performs poorly on visual-centric
GUI tasks, especially for high-level planning.

通过视频 GUI 评估可视化导向的图形用户界面 (GUI) 任务上 GUI 助手的表现，并发现当前最先进的大型多模态模型 GPT4o 在高级规划方面表现不佳。