Macros are building block tasks of our everyday smartphone activity (e.g.,
"login", or "booking a flight"). Effectively extracting macros is important for
understanding mobile interaction and enabling task automation. These macros are
however difficult to extract at scale as they can be comprised of multiple
steps yet hidden within programmatic components of the app. In this paper, we
introduce a novel approach based on Large Language Models (LLMs) to
automatically extract semantically meaningful macros from both random and
user-curated mobile interaction traces. The macros produced by our approach are
automatically tagged with natural language descriptions and are fully
executable. To examine the quality of extraction, we conduct multiple studies,
including user evaluation, comparative analysis against human-curated tasks,
and automatic execution of these macros. These experiments and analyses show
the effectiveness of our approach and the usefulness of extracted macros in
various downstream applications.

本文介绍了一种基于大型语言模型的新方法，可以自动从随机和用户策划的移动交互痕迹中提取具有语义含义的宏，并带有自然语言描述和完全可执行的标签。通过用户评估、与人工策划任务的比较分析以及这些宏的自动执行，我们对提取的质量进行了多项研究和分析，展示了我们方法的有效性和提取宏用于各种下游应用的实用性。

大规模互动追踪的自动宏挖掘

Automatic Macro Mining from Interaction Traces at Scale

Conversational agents show the promise to allow users to interact with mobile
devices using language. However, to perform diverse UI tasks with natural
language, developers typically need to create separate datasets and models for
each specific task, which is expensive and effort-consuming. Recently,
pre-trained large language models (LLMs) have been shown capable of
generalizing to various downstream tasks when prompted with a handful of
examples from the target task. This paper investigates the feasibility of
enabling versatile conversational interactions with mobile UIs using a single
LLM. We designed prompting techniques to adapt an LLM to mobile UIs. We
experimented with four important modeling tasks that address various scenarios
in conversational interaction. Our method achieved competitive performance on
these challenging tasks without requiring dedicated datasets and training,
offering a lightweight and generalizable approach to enable language-based
mobile interaction.

本文旨在通过预训练大型语言模型及相应的提示技术，实现轻量级、可通用的基于自然语言的手机交互，解决开发人员需要针对每项具体任务创建独立数据集和模型的成本和劳动力问题。研究表明，针对移动 UI 设计的四项重要建模任务中，我们的方法在不需要专用数据集和训练的情况下，取得了令人满意的竞争成绩。