The advent of Multimodal LLMs has significantly enhanced image OCR
recognition capabilities, making GUI automation a viable reality for increasing
efficiency in digital tasks. One fundamental aspect of developing a GUI
automation system is understanding primitive GUI actions. This comprehension is
crucial as it enables agents to learn from user demonstrations, an essential
element of automation. To rigorously evaluate such capabilities, we developed a
video captioning benchmark for GUI actions, comprising 4,189 diverse video
captioning samples. This task presents unique challenges compared to natural
scene video captioning: 1) GUI screenshots typically contain denser information
than natural scenes, and 2) events within GUIs are subtler and occur more
rapidly, requiring precise attention to the appropriate time span and spatial
region for accurate understanding. To address these challenges, we introduce
our GUI action dataset \textbf{Act2Cap} as well as a simple yet effective
framework, \textbf{GUI Narrator}, for GUI video captioning that utilizes the
cursor as a visual prompt to enhance the interpretation of high-resolution
screenshots. Specifically, a cursor detector is trained on our dataset, and a
multimodal LLM model with mechanisms for selecting keyframes and key regions
generates the captions. Experimental results indicate that even for today's
most advanced multimodal models, such as GPT-4o, the task remains highly
challenging. Additionally, our evaluations show that our strategy effectively
enhances model performance, whether integrated into the fine-tuning of
open-source models or employed as a prompting strategy in closed-source models.

通过引入语言与视觉多模态学习模型，研究提高图像 OCR 识别能力并实现 GUI 自动化，开发了适用于 GUI 动作的视频字幕基准以及利用光标作为视觉提示的 GUI 视频字幕框架，提升了高分辨率截图的解释性。实验结果表明，即使对于如 GPT-4o 等最先进的多模态模型，该任务仍具有很高的挑战性，并且我们的策略可以有效增强模型性能。

图形用户界面操作叙述器：行为发生的地点和时间是什么？

GUI Action Narrator: Where and When Did That Action Take Place?

Large language models (LLMs) have shown remarkable potential as human-like
autonomous language agents to interact with real-world environments, especially
for graphical user interface (GUI) automation. However, those GUI agents
require comprehensive cognition ability including exhaustive perception and
reliable action response. We propose \underline{Co}mprehensive
\underline{Co}gnitive LLM \underline{Agent}, CoCo-Agent, with two novel
approaches, comprehensive environment perception (CEP) and conditional action
prediction (CAP), to systematically improve the GUI automation performance.
First, CEP facilitates the GUI perception through different aspects and
granularity, including screenshots and complementary detailed layouts for the
visual channel and historical actions for the textual channel. Second, CAP
decomposes the action prediction into sub-problems: action type prediction and
action target conditioned on the action type. With our technical design, our
agent achieves new state-of-the-art performance on AITW and META-GUI
benchmarks, showing promising abilities in realistic scenarios.

我们提出了一种全面认知的大型语言模型代理，CoCo-Agent，通过全面环境感知和条件化行动预测的两种新方法系统地提高了图形用户界面自动化性能。我们的代理在 AITW 和 META-GUI 基准测试中取得了最新的最佳表现，展示了在现实场景中的巨大潜力。