New smartphone users have difficulty engaging with it and often use only a
limited set of features like calling and messaging. These users are hesitant to
explore using the smartphone and rely on experienced users to teach them how to
use the phone. However, experienced users are not always around to guide them.
To help new users learn how to use the phone on their own, we propose a natural
language based instruction following agent that operates over the UI and shows
the user how to perform various tasks. Common how-to questions, such as "How to
block calls from unknown numbers?", are documented on support sites with a
sequence of steps in natural language describing what the user should do. We
parse these steps using Large Language Models (LLMs) and generate macros that
can be executed on-device when the user asks a query. To evaluate this agent,
we introduce UGIF-DataSet, a multi-lingual, multi-modal UI grounded dataset for
step-by-step task completion on the smartphone. It contains 523 natural
language instructions with paired sequences of multilingual UI screens and
actions that show how to execute the task in eight languages. We compare the
performance of different large language models including PaLM, GPT3, etc. and
find that the end-to-end task completion success rate is 48% for English UI but
the performance drops to 32% for non-English languages. We analyse the common
failure modes of existing models on this task and point out areas for
improvement.

为帮助新手学习如何自己使用智能手机，我们提出了一种基于自然语言的指导代理，然后使用大型语言模型解析自然语言说明并生成可以在设备上执行的宏。我们介绍了一个多语言、多模式的 UI 基础数据集 UGIF-DataSet，我们对不同的大型语言模型的性能进行比较，发现任务完成成功率为英语 UI 为 48%，非英语语言的性能下降到 32%。

UGIF: UI 基础指令跟踪

UGIF: UI Grounded Instruction Following

Mental health conditions remain underdiagnosed even in countries with common
access to advanced medical care. The ability to accurately and efficiently
predict mood from easily collectible data has several important implications
for the early detection, intervention, and treatment of mental health
disorders. One promising data source to help monitor human behavior is daily
smartphone usage. However, care must be taken to summarize behaviors without
identifying the user through personal (e.g., personally identifiable
information) or protected (e.g., race, gender) attributes. In this paper, we
study behavioral markers of daily mood using a recent dataset of mobile
behaviors from adolescent populations at high risk of suicidal behaviors. Using
computational models, we find that language and multimodal representations of
mobile typed text (spanning typed characters, words, keystroke timings, and app
usage) are predictive of daily mood. However, we find that models trained to
predict mood often also capture private user identities in their intermediate
representations. To tackle this problem, we evaluate approaches that obfuscate
user identity while remaining predictive. By combining multimodal
representations with privacy-preserving learning, we are able to push forward
the performance-privacy frontier.

通过对高自杀风险人群进行研究，本文从语言和移动文本的多模态表示等方面分析了手机使用行为与情绪的关联，并提出一种保护隐私的深度学习方法来提高智能手机数据的使用价值。

从移动数据中学习语言和多模态隐私保护的情绪标记

Learning Language and Multimodal Privacy-Preserving Markers of Mood from  Mobile Data

With the rapid growth in smartphone usage, more organizations begin to focus
on providing better services for mobile users. User identification can help
these organizations to identify their customers and then cater services that
have been customized for them. Currently, the use of cookies is the most common
form to identify users. However, cookies are not easily transportable (e.g.,
when a user uses a different login account, cookies do not follow the user).
This limitation motivates the need to use behavior biometric for user
identification. In this paper, we propose DEEPSERVICE, a new technique that can
identify mobile users based on user's keystroke information captured by a
special keyboard or web browser. Our evaluation results indicate that
DEEPSERVICE is highly accurate in identifying mobile users (over 93% accuracy).
The technique is also efficient and only takes less than 1 ms to perform
identification.

本文提出了一种新技术 DEEPSERVICE 来识别移动用户，该技术基于用户的击键信息进行识别，准确率高达 93％以上，并且只需不到 1 毫秒的时间。