Recent advances in foundation models, particularly Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), facilitate intelligent agents being capable of performing complex tasks. By leveraging the ability of (M)LLMs to process and interpret Graphical User Interfaces (GUIs), these agents can autonomously execute user instructions by simulating human-like interactions such as clicking and typing. This survey consolidates recent research on (M)LLM-based GUI agents, highlighting key innovations in data, frameworks, and applications. We begin by discussing representative datasets and benchmarks. Next, we summarize a unified framework that captures the essential components used in prior research, accompanied by a taxonomy. Additionally, we explore commercial applications of (M)LLM-based GUI agents. Drawing from existing work, we identify several key challenges and propose future research directions. We hope this paper will inspire further developments in the field of (M)LLM-based GUI agents.

本研究针对基础模型尤其是大型语言模型在图形用户界面代理中的应用进行了综述，填补了相关文献中对不同数据集、框架和应用的整合性分析的空白。文章提出了一种统一框架，展示了之前研究中的核心组件和分类法，并识别出关键挑战与未来研究方向，旨在推动该领域的进一步发展。

基础模型驱动的图形用户界面代理：全面综述