Addressing the challenge of a digital assistant capable of executing a wide
array of user tasks, our research focuses on the realm of instruction-based
mobile device control. We leverage recent advancements in large language models
(LLMs) and present a visual language model (VLM) that can fulfill diverse tasks
on mobile devices. Our model functions by interacting solely with the user
interface (UI). It uses the visual input from the device screen and mimics
human-like interactions, encompassing gestures such as tapping and swiping.
This generality in the input and output space allows our agent to interact with
any application on the device. Unlike previous methods, our model operates not
only on a single screen image but on vision-language sentences created from
sequences of past screenshots along with corresponding actions. Evaluating our
method on the challenging Android in the Wild benchmark demonstrates its
promising efficacy and potential.

利用大型语言模型和视觉语言模型，我们的研究致力于解决数字助理执行各种用户任务的挑战，特别是在基于指令的移动设备控制领域。通过与用户界面进行交互，我们的模型利用设备屏幕的视觉输入并模拟人类般的交互，包括点击和滑动等手势。这种输入和输出空间的广泛适用性使得我们的代理能够与设备上的任何应用程序进行交互。与以往方法不同的是，我们的模型不仅仅操作单个屏幕图像，还通过过去截图序列和相应操作生成视觉语言句子。在具有挑战性的 “Android in the Wild” 基准测试中评估我们的方法表明其有效性和潜力。