Conversational agents show the promise to allow users to interact with mobile
devices using language. However, to perform diverse UI tasks with natural
language, developers typically need to create separate datasets and models for
each specific task, which is expensive and effort-consuming. Recently,
pre-trained large language models (LLMs) have been shown capable of
generalizing to various downstream tasks when prompted with a handful of
examples from the target task. This paper investigates the feasibility of
enabling versatile conversational interactions with mobile UIs using a single
LLM. We designed prompting techniques to adapt an LLM to mobile UIs. We
experimented with four important modeling tasks that address various scenarios
in conversational interaction. Our method achieved competitive performance on
these challenging tasks without requiring dedicated datasets and training,
offering a lightweight and generalizable approach to enable language-based
mobile interaction.

本文旨在通过预训练大型语言模型及相应的提示技术，实现轻量级、可通用的基于自然语言的手机交互，解决开发人员需要针对每项具体任务创建独立数据集和模型的成本和劳动力问题。研究表明，针对移动 UI 设计的四项重要建模任务中，我们的方法在不需要专用数据集和训练的情况下，取得了令人满意的竞争成绩。

利用大型语言模型实现移动界面的交互式对话功能

Enabling Conversational Interaction with Mobile UI using Large Language Models

Predicting human performance in interaction tasks allows designers or
developers to understand the expected performance of a target interface without
actually testing it with real users. In this work, we present a deep neural net
to model and predict human performance in performing a sequence of UI tasks. In
particular, we focus on a dominant class of tasks, i.e., target selection from
a vertical list or menu. We experimented with our deep neural net using a
public dataset collected from a desktop laboratory environment and a dataset
collected from hundreds of touchscreen smartphone users via crowdsourcing. Our
model significantly outperformed previous methods on these datasets.
Importantly, our method, as a deep model, can easily incorporate additional UI
attributes such as visual appearance and content semantics without changing
model architectures. By understanding about how a deep learning model learns
from human behaviors, our approach can be seen as a vehicle to discover new
patterns about human behaviors to advance analytical modeling.

本文介绍了一种使用深度神经网络预测人类执行 UI 任务表现的方法，该方法可以轻松地加入其他 UI 属性以便于分析建模，同时能够发现人类行为的新模式