Voice assistants help users make phone calls, send messages, create events,
navigate, and do a lot more. However, assistants have limited capacity to
understand their users' context. In this work, we aim to take a step in this
direction. Our work dives into a new experience for users to refer to phone
numbers, addresses, email addresses, URLs, and dates on their phone screens.
Our focus lies in reference understanding, which becomes particularly
interesting when multiple similar texts are present on screen, similar to
visual grounding. We collect a dataset and propose a lightweight
general-purpose model for this novel experience. Due to the high cost of
consuming pixels directly, our system is designed to rely on the extracted text
from the UI. Our model is modular, thus offering flexibility, improved
interpretability, and efficient runtime memory utilization.

本文提出一种基于模块化模型和从 UI 中提取的文本的新型体验，旨在改善助手理解用户语境的能力，涉及语音助手，参考理解，数据集等。

使用语音助手引用屏幕文字

Referring to Screen Texts with Voice Assistants

Optimization in multi-task learning (MTL) is more challenging than
single-task learning (STL), as the gradient from different tasks can be
contradictory. When tasks are related, it can be beneficial to share some
parameters among them (cooperation). However, some tasks require additional
parameters with expertise in a specific type of data or discrimination
(specialization). To address the MTL challenge, we propose Mod-Squad, a new
model that is Modularized into groups of experts (a 'Squad'). This structure
allows us to formalize cooperation and specialization as the process of
matching experts and tasks. We optimize this matching process during the
training of a single model. Specifically, we incorporate mixture of experts
(MoE) layers into a transformer model, with a new loss that incorporates the
mutual dependence between tasks and experts. As a result, only a small set of
experts are activated for each task. This prevents the sharing of the entire
backbone model between all tasks, which strengthens the model, especially when
the training set size and the number of tasks scale up. More interestingly, for
each task, we can extract the small set of experts as a standalone model that
maintains the same performance as the large model. Extensive experiments on the
Taskonomy dataset with 13 vision tasks and the PASCAL-Context dataset with 5
vision tasks show the superiority of our approach.

该研究提出了 Mod-Squad 模型，使用模块化结构将组内的专家与任务相匹配，并在训练过程中优化匹配过程以对抗多任务学习的挑战。实验表明本方法在任务数量和训练集规模扩大时表现强于其他方法。