Autonomous user interface (UI) agents aim to facilitate task automation by
interacting with the user interface without manual intervention. Recent studies
have investigated eliciting the capabilities of large language models (LLMs)
for effective engagement in diverse environments. To align with the
input-output requirement of LLMs, existing approaches are developed under a
sandbox setting where they rely on external tools and application-specific APIs
to parse the environment into textual elements and interpret the predicted
actions. Consequently, those approaches often grapple with inference
inefficiency and error propagation risks. To mitigate the challenges, we
introduce Auto-UI, a multimodal solution that directly interacts with the
interface, bypassing the need for environment parsing or reliance on
application-dependent APIs. Moreover, we propose a chain-of-action technique --
leveraging a series of intermediate previous action histories and future action
plans -- to help the agent decide what action to execute. We evaluate our
approach on a new device-control benchmark AITW with 30K unique instructions,
spanning multi-step tasks such as application operation, web searching, and web
shopping. Experimental results show that Auto-UI achieves state-of-the-art
performance with an action type prediction accuracy of 90% and an overall
action success rate of 74%. Code is publicly available at
this https URL

自主用户界面 (UI) 代理旨在通过与用户界面的自动交互来促进任务自动化。为了对齐于大型语言模型 (LLMs) 的输入 - 输出需求，现有方法在沙盒环境下开发，依赖外部工具和应用特定的 API 将环境解析为文本元素并解释预测的动作。为了减轻这些挑战，我们引入了 Auto-UI，这是一个多模态的解决方案，直接与界面交互，无需环境解析或依赖应用程序相关的 API。此外，我们提出了一种链式动作技术 —— 利用一系列中间的先前动作历史和未来动作计划 —— 以帮助代理决定执行什么动作。我们在一个新的设备控制基准 AITW 上评估了我们的方法，该基准包含 30K 个独特的指令，涵盖应用操作、网络搜索和网购等多步任务。实验结果显示，Auto-UI 实现了 90% 的动作类型预测准确率和 74% 的整体动作成功率。