Developing autonomous agents for mobile devices can significantly enhance
user interactions by offering increased efficiency and accessibility. However,
despite the growing interest in mobile device control agents, the absence of a
commonly adopted benchmark makes it challenging to quantify scientific progress
in this area. In this work, we introduce B-MoCA: a novel benchmark designed
specifically for evaluating mobile device control agents. To create a realistic
benchmark, we develop B-MoCA based on the Android operating system and define
60 common daily tasks. Importantly, we incorporate a randomization feature that
changes various aspects of mobile devices, including user interface layouts and
language settings, to assess generalization performance. We benchmark diverse
agents, including agents employing large language models (LLMs) or multi-modal
LLMs as well as agents trained from scratch using human expert demonstrations.
While these agents demonstrate proficiency in executing straightforward tasks,
their poor performance on complex tasks highlights significant opportunities
for future research to enhance their effectiveness. Our source code is publicly
available at this https URL

通过引入 B-MoCA 基准测试，本研究针对移动设备控制代理开发自主代理，提高用户交互的效率和可访问性。在 Android 操作系统上创建了一个真实的基准测试，定义了 60 项常见日常任务，并通过随机化功能来评估代理的泛化性能。尽管代理们在执行简单任务方面展示出熟练性，但在复杂任务上的低效性凸显了未来研究改进其效果的重要机遇。