The advent of large language models (LLMs) has opened up new opportunities in
the field of mobile task automation. Their superior language understanding and
reasoning capabilities allow users to automate complex and repetitive tasks.
However, due to the inherent unreliability and high operational cost of LLMs,
their practical applicability is quite limited. To address these issues, this
paper introduces MemoDroid, an innovative LLM-based mobile task automator
enhanced with a unique app memory. MemoDroid emulates the cognitive process of
humans interacting with a mobile app -- explore, select, derive, and recall.
This approach allows for a more precise and efficient learning of a task's
procedure by breaking it down into smaller, modular components that can be
re-used, re-arranged, and adapted for various objectives. We implement
MemoDroid using online LLMs services (GPT-3.5 and GPT-4) and evaluate its
performance on 50 unique mobile tasks across 5 widely used mobile apps. The
results indicate that MemoDroid can adapt learned tasks to varying contexts
with 100% accuracy and reduces their latency and cost by 69.22% and 77.36%
compared to a GPT-4 powered baseline.

MemoDroid 是一种基于大型语言模型的创新移动任务自动化器，通过模拟人类与移动应用程序的认知过程（探索、选择、推导和回忆），将任务分解为可重用、可重新排列和适应于各种目标的模块化组件，实现对任务过程的精确和高效学习。使用在线语言模型服务（GPT-3.5 和 GPT-4）实现 MemoDroid，并在 5 个广泛使用的移动应用程序上对 50 个独特的移动任务进行评估。结果表明，与 GPT-4 驱动的基线相比，MemoDroid 能够以 100％的准确性适应不同的上下文，将任务的延迟和成本分别降低了 69.22％和 77.36％。

探索、选择、派生和召回：为移动任务自动化增添类似人类记忆的 LLM

Explore, Select, Derive, and Recall: Augmenting LLM with Human-like  Memory for Mobile Task Automation

Mobile task automation is an attractive technique that aims to enable
voice-based hands-free user interaction with smartphones. However, existing
approaches suffer from poor scalability due to the limited language
understanding ability and the non-trivial manual efforts required from
developers or end-users. The recent advance of large language models (LLMs) in
language understanding and reasoning inspires us to rethink the problem from a
model-centric perspective, where task preparation, comprehension, and execution
are handled by a unified language model. In this work, we introduce AutoDroid,
a mobile task automation system that can handle arbitrary tasks on any Android
application without manual efforts. The key insight is to combine the
commonsense knowledge of LLMs and domain-specific knowledge of apps through
automated dynamic analysis. The main components include a functionality-aware
UI representation method that bridges the UI with the LLM, exploration-based
memory injection techniques that augment the app-specific domain knowledge of
LLM, and a multi-granularity query optimization module that reduces the cost of
model inference. We integrate AutoDroid with off-the-shelf LLMs including
online GPT-4/GPT-3.5 and on-device Vicuna, and evaluate its performance on a
new benchmark for memory-augmented Android task automation with 158 common
tasks. The results demonstrated that AutoDroid is able to precisely generate
actions with an accuracy of 90.9%, and complete tasks with a success rate of
71.3%, outperforming the GPT-4-powered baselines by 36.4% and 39.7%. The demo,
benchmark suites, and source code of AutoDroid will be released at
this https URL

AutoDroid 是一种能够在 Android 应用上自动执行任意任务且无需人工努力的移动任务自动化系统，通过自动化的动态分析将 LLMs 的常识知识与应用程序的领域特定知识相结合，实现了功能感知的 UI 表示方法、基于探索的内存注入技术以及多粒度查询优化模块等主要组件，表现出了优于 GPT-4 的效果。