Utilizing Graphic User Interface (GUI) for human-computer interaction is
essential for accessing a wide range of digital tools. Recent advancements in
Vision Language Models (VLMs) highlight the compelling potential to develop
versatile agents to help humans finish GUI navigation tasks. However, current
VLMs are challenged in terms of fundamental abilities (OCR and grounding) and
GUI knowledge (the functions and control methods of GUI elements), preventing
them from becoming practical GUI agents. To solve these challenges, we
contribute GUICourse, a suite of datasets to train visual-based GUI agents from
general VLMs. First, we introduce the GUIEnv dataset to strengthen the OCR and
grounding capabilities of VLMs. Then, we introduce the GUIAct and GUIChat
datasets to enrich their knowledge of GUI components and interactions.
Experiments demonstrate that our GUI agents have better performance on common
GUI tasks than their baseline VLMs. Even the small-size GUI agent (with 3.1B
parameters) can still work well on single-step and multi-step GUI tasks.
Finally, we analyze the different varieties in the training stage of this agent
by ablation study. Our source codes and datasets are released at
this https URL

使用图形用户界面（Graphic User Interface）进行人机交互是访问各种数字工具的基本要素，在最近的视觉语言模型（Vision Language Models，VLMs）的发展中，发现了其潜力，可以开发多功能代理人来帮助人类完成图形用户界面导航任务，然而，现有的视觉语言模型在基本能力（OCR 和定位）和图形用户界面知识（图形用户界面元素的功能和控制方法）方面面临挑战，从而无法成为实用的图形用户界面代理人，为了解决这些挑战，我们提出了 GUICourse，一套用于从通用视觉语言模型中训练基于视觉的图形用户界面代理人的数据集，首先，我们引入了 GUIEnv 数据集来增强视觉语言模型的 OCR 和定位能力，然后，我们引入了 GUIAct 和 GUIChat 数据集来丰富其图形用户界面组件和交互的知识，实验证明，我们的图形用户界面代理人在常见的图形用户界面任务上比基准视觉语言模型具有更好的性能，即使是小型的图形用户界面代理人（具有 31 亿个参数），仍然能够在单步和多步图形用户界面任务上表现良好，最后，我们通过消融研究分析了训练阶段中这个代理人之间的不同变化，我们的源代码和数据集已在此 https URL 上发布。