Existing Large Language Models (LLM) can invoke a variety of tools and APIs
to complete complex tasks. The computer, as the most powerful and universal
tool, could potentially be controlled directly by a trained LLM agent. Powered
by the computer, we can hopefully build a more generalized agent to assist
humans in various daily digital works. In this paper, we construct an
environment for a Vision Language Model (VLM) agent to interact with a real
computer screen. Within this environment, the agent can observe screenshots and
manipulate the Graphics User Interface (GUI) by outputting mouse and keyboard
actions. We also design an automated control pipeline that includes planning,
acting, and reflecting phases, guiding the agent to continuously interact with
the environment and complete multi-step tasks. Additionally, we construct the
ScreenAgent Dataset, which collects screenshots and action sequences when
completing a variety of daily computer tasks. Finally, we trained a model,
ScreenAgent, which achieved computer control capabilities comparable to GPT-4V
and demonstrated more precise UI positioning capabilities. Our attempts could
inspire further research on building a generalist LLM agent. The code is
available at https://github.com/niuzaisheng/ScreenAgent.

本文构建了一个用于视觉语言模型（VLM）代理与真实计算机屏幕交互的环境，在该环境中，代理可以观察屏幕截图并通过输出鼠标和键盘动作操纵图形用户界面（GUI）。我们还设计了一个自动控制流程，包括规划、执行和反思阶段，指导代理不断与环境交互并完成多步骤任务。此外，我们构建了 ScreenAgent 数据集，收集完成多种日常计算机任务时的屏幕截图和动作序列。最后，我们训练了一个名为 ScreenAgent 的模型，其实现了与 GPT-4V 相媲美的计算机控制能力，并展示了更精确的用户界面定位能力。这些尝试有望进一步激发构建通用性 LLM 代理的研究。源代码可在 https://github.com/niuzaisheng/ScreenAgent 获取。