Recently, Multimodal Large Language Models (MLLMs) have been used as agents
to control keyboard and mouse inputs by directly perceiving the Graphical User
Interface (GUI) and generating corresponding code. However, current agents
primarily exhibit excellent understanding capabilities in static environments
and are predominantly applied in relatively simple domains, such as Web or
mobile interfaces. We argue that a robust GUI agent should be capable of
perceiving temporal information on the GUI, including dynamic Web content and
multi-step tasks. Additionally, it should possess a comprehensive understanding
of various GUI scenarios, including desktop software and multi-window
interactions. To this end, this paper introduces a new dataset, termed
GUI-World, which features meticulously crafted Human-MLLM annotations,
extensively covering six GUI scenarios and eight types of GUI-oriented
questions in three formats. We evaluate the capabilities of current
state-of-the-art MLLMs, including ImageLLMs and VideoLLMs, in understanding
various types of GUI content, especially dynamic and sequential content. Our
findings reveal that ImageLLMs struggle with dynamic GUI content without
manually annotated keyframes or operation history. On the other hand, VideoLLMs
fall short in all GUI-oriented tasks given the sparse GUI video dataset. Based
on GUI-World, we take the initial step of leveraging a fine-tuned VideoLLM as a
GUI agent, demonstrating an improved understanding of various GUI tasks.
However, due to the limitations in the performance of base LLMs, we conclude
that using VideoLLMs as GUI agents remains a significant challenge. We believe
our work provides valuable insights for future research in dynamic GUI content
understanding. The code and dataset are publicly available at our project
homepage: this https URL

该研究论文介绍了一个新的数据集，名为 GUI-World，包含六种 GUI 场景和八种 GUI 问题类型的人工 - MLLM 注释，评估了当前最先进的 MLLMs（包括 ImageLLMs 和 VideoLLMs）在理解各种 GUI 内容方面的能力，发现 ImageLLMs 在没有手动注释的关键帧或操作历史的情况下难以处理动态 GUI 内容，而 VideoLLMs 在 GUI 视频数据集稀缺的情况下在所有 GUI 任务中表现不佳。然而，作者通过利用 Fine-tuned VideoLLM 作为 GUI agent 迈出了一步，展示了对各种 GUI 任务的改进理解，但由于基础 LLMs 的性能限制，使用 VideoLLMs 作为 GUI agent 仍然是一个重大挑战，研究为未来动态 GUI 内容理解的研究提供了有价值的见解。