Much of the previous work towards digital agents for graphical user
interfaces (GUIs) has relied on text-based representations (derived from HTML
or other structured data sources), which are not always readily available.
These input representations have been often coupled with custom, task-specific
action spaces. This paper focuses on creating agents that interact with the
digital world using the same conceptual interface that humans commonly use --
via pixel-based screenshots and a generic action space corresponding to
keyboard and mouse actions. Building upon recent progress in pixel-based
pretraining, we show, for the first time, that it is possible for such agents
to outperform human crowdworkers on the MiniWob++ benchmark of GUI-based
instruction following tasks.

本研究提出一种使用基于像素的屏幕截图和一般的鼠标和键盘动作空间来模拟人与数字世界互动的智能代理。通过依赖于最近的像素预训练技术，该代理在 GUI 任务的 MinWob ++ 基准测试中表现出比人类众包工人更好的性能。