Graphical User Interface (GUI) automation holds significant promise for
enhancing human productivity by assisting with computer tasks. Existing task
formulations primarily focus on simple tasks that can be specified by a single,
language-only instruction, such as "Insert a new slide." In this work, we
introduce VideoGUI, a novel multi-modal benchmark designed to evaluate GUI
assistants on visual-centric GUI tasks. Sourced from high-quality web
instructional videos, our benchmark focuses on tasks involving professional and
novel software (e.g., Adobe Photoshop or Stable Diffusion WebUI) and complex
activities (e.g., video editing). VideoGUI evaluates GUI assistants through a
hierarchical process, allowing for identification of the specific levels at
which they may fail: (i) high-level planning: reconstruct procedural subtasks
from visual conditions without language descriptions; (ii) middle-level
planning: generate sequences of precise action narrations based on visual state
(i.e., screenshot) and goals; (iii) atomic action execution: perform specific
actions such as accurately clicking designated elements. For each level, we
design evaluation metrics across individual dimensions to provide clear
signals, such as individual performance in clicking, dragging, typing, and
scrolling for atomic action execution. Our evaluation on VideoGUI reveals that
even the SoTA large multimodal model GPT4o performs poorly on visual-centric
GUI tasks, especially for high-level planning.

通过视频 GUI 评估可视化导向的图形用户界面 (GUI) 任务上 GUI 助手的表现，并发现当前最先进的大型多模态模型 GPT4o 在高级规划方面表现不佳。

VideoGUI: 从教学视频中的 GUI 自动化基准

VideoGUI: A Benchmark for GUI Automation from Instructional Videos

Graphical User Interface (GUI) automation holds significant promise for
assisting users with complex tasks, thereby boosting human productivity.
Existing works leveraging Large Language Model (LLM) or LLM-based AI agents
have shown capabilities in automating tasks on Android and Web platforms.
However, these tasks are primarily aimed at simple device usage and
entertainment operations. This paper presents a novel benchmark, AssistGUI, to
evaluate whether models are capable of manipulating the mouse and keyboard on
the Windows platform in response to user-requested tasks. We carefully
collected a set of 100 tasks from nine widely-used software applications, such
as, After Effects and MS Word, each accompanied by the necessary project files
for better evaluation. Moreover, we propose an advanced Actor-Critic Embodied
Agent framework, which incorporates a sophisticated GUI parser driven by an
LLM-agent and an enhanced reasoning mechanism adept at handling lengthy
procedural tasks. Our experimental results reveal that our GUI Parser and
Reasoning mechanism outshine existing methods in performance. Nevertheless, the
potential remains substantial, with the best model attaining only a 46% success
rate on our benchmark. We conclude with a thorough analysis of the current
methods' limitations, setting the stage for future breakthroughs in this
domain.

使用大型语言模型的图形用户界面（GUI）自动化在处理复杂任务方面有很大潜力。本论文介绍了一个新的基准测试框架 AssistGUI，评估模型是否能够响应用户请求的任务，在 Windows 平台上操纵鼠标和键盘。实验结果显示，我们的 GUI Parser 和 Reasoning 机制在性能上优于现有方法，但在基准测试中，最佳模型的成功率仅为 46％。通过对当前方法的限制进行彻底分析，为未来在该领域取得突破奠定基础。