Retrieval is a widely adopted approach for improving language models
leveraging external information. As the field moves towards multi-modal large
language models, it is important to extend the pure text based methods to
incorporate other modalities in retrieval as well for applications across the
wide spectrum of machine learning tasks and data types. In this work, we
propose multi-modal retrieval with two approaches: kNN-LM and cross-attention
techniques. We demonstrate the effectiveness of our retrieval approaches
empirically by applying them to automatic speech recognition tasks with access
to external information. Under this setting, we show that speech-based
multi-modal retrieval outperforms text based retrieval, and yields up to 50 %
improvement in word error rate over the multi-modal language model baseline.
Furthermore, we achieve state-of-the-art recognition results on the
Spoken-Squad question answering dataset.

提出 kNN-LM 和跨注意力技术两种方法的多模态检索，证明了其在语音识别任务上优于基于文本的检索方法，对多模态语言模型基准线具有最高水平的识别结果。

基于大型语言模型的多模态检索用于语音识别

Multi-Modal Retrieval For Large Language Model Based Speech Recognition

Current activity tracking technologies are largely trained on younger adults'
data, which can lead to solutions that are not well-suited for older adults. To
build activity trackers for older adults, it is crucial to collect training
data with them. To this end, we examine the feasibility and challenges with
older adults in collecting activity labels by leveraging speech. Specifically,
we built MyMove, a speech-based smartwatch app to facilitate the in-situ
labeling with a low capture burden. We conducted a 7-day deployment study,
where 13 older adults collected their activity labels and smartwatch sensor
data, while wearing a thigh-worn activity monitor. Participants were highly
engaged, capturing 1,224 verbal reports in total. We extracted 1,885 activities
with corresponding effort level and timespan, and examined the usefulness of
these reports as activity labels. We discuss the implications of our approach
and the collected dataset in supporting older adults through personalized
activity tracking technologies.

研究以语音为基础的智能手表应用程序 MyMove，通过老年人接收活动标签，可支持个性化活动跟踪技术。