Existing Large Language Models (LLM) can invoke a variety of tools and APIs
to complete complex tasks. The computer, as the most powerful and universal
tool, could potentially be controlled directly by a trained LLM agent. Powered
by the computer, we can hopefully build a more generalized agent to assist
humans in various daily digital works. In this paper, we construct an
environment for a Vision Language Model (VLM) agent to interact with a real
computer screen. Within this environment, the agent can observe screenshots and
manipulate the Graphics User Interface (GUI) by outputting mouse and keyboard
actions. We also design an automated control pipeline that includes planning,
acting, and reflecting phases, guiding the agent to continuously interact with
the environment and complete multi-step tasks. Additionally, we construct the
ScreenAgent Dataset, which collects screenshots and action sequences when
completing a variety of daily computer tasks. Finally, we trained a model,
ScreenAgent, which achieved computer control capabilities comparable to GPT-4V
and demonstrated more precise UI positioning capabilities. Our attempts could
inspire further research on building a generalist LLM agent. The code is
available at https://github.com/niuzaisheng/ScreenAgent.

本文构建了一个用于视觉语言模型（VLM）代理与真实计算机屏幕交互的环境，在该环境中，代理可以观察屏幕截图并通过输出鼠标和键盘动作操纵图形用户界面（GUI）。我们还设计了一个自动控制流程，包括规划、执行和反思阶段，指导代理不断与环境交互并完成多步骤任务。此外，我们构建了 ScreenAgent 数据集，收集完成多种日常计算机任务时的屏幕截图和动作序列。最后，我们训练了一个名为 ScreenAgent 的模型，其实现了与 GPT-4V 相媲美的计算机控制能力，并展示了更精确的用户界面定位能力。这些尝试有望进一步激发构建通用性 LLM 代理的研究。源代码可在 https://github.com/niuzaisheng/ScreenAgent 获取。

ScreenAgent：基于视觉语言模型的计算机控制代理

ScreenAgent: A Vision Language Model-driven Computer Control Agent

It would be useful for machines to use computers as humans do so that they
can aid us in everyday tasks. This is a setting in which there is also the
potential to leverage large-scale expert demonstrations and human judgements of
interactive behaviour, which are two ingredients that have driven much recent
success in AI. Here we investigate the setting of computer control using
keyboard and mouse, with goals specified via natural language. Instead of
focusing on hand-designed curricula and specialized action spaces, we focus on
developing a scalable method centered on reinforcement learning combined with
behavioural priors informed by actual human-computer interactions. We achieve
state-of-the-art and human-level mean performance across all tasks within the
MiniWob++ benchmark, a challenging suite of computer control problems, and find
strong evidence of cross-task transfer. These results demonstrate the
usefulness of a unified human-agent interface when training machines to use
computers. Altogether our results suggest a formula for achieving competency
beyond MiniWob++ and towards controlling computers, in general, as a human
would.

该研究使用强化学习和行为先验相结合的方法，以自然语言指定目标，实现了最新的并具有人类水平的表现，并证明了在训练机器使用计算机时，统一的人机界面是非常有用的。