We recently developed SLM, a joint speech and language model, which fuses a
pretrained foundational speech model and a large language model (LLM), while
preserving the in-context learning capability intrinsic to the pretrained LLM.
In this paper, we apply SLM to speech dialog applications where the dialog
states are inferred directly from the audio signal.
Task-oriented dialogs often contain domain-specific entities, i.e.,
restaurants, hotels, train stations, and city names, which are difficult to
recognize, however, critical for the downstream applications. Inspired by the
RAG (retrieval-augmented generation) paradigm, we propose a retrieval augmented
SLM (ReSLM) that overcomes this weakness. We first train a speech retriever to
retrieve text entities mentioned in the audio. The retrieved entities are then
added as text inputs to the underlying SLM to bias model predictions. We
evaluated ReSLM on speech MultiWoz task (DSTC-11 challenge), and found that
this retrieval augmentation boosts model performance, achieving joint goal
accuracy (38.6% vs 32.7%), slot error rate (20.6% vs 24.8%) and ASR word error
rate (5.5% vs 6.7%). While demonstrated on dialog state tracking, our approach
is broadly applicable to other speech tasks requiring contextual information or
domain-specific entities, such as contextual ASR with biasing capability.

我们将先前开发的 SLM 模型应用于语音对话应用中，其中对话状态直接从音频信号中推断，通过检索增强的 SLM（ReSLM）改善了识别领域特定实体的困难，提升了模型性能，特别在对话状态跟踪和具有偏置能力的语音自动识别方面具有广泛适用性。

检索增强的端到端口语对话模型

Retrieval Augmented End-to-End Spoken Dialog Models

Contextual ASR, which takes a list of bias terms as input along with audio,
has drawn recent interest as ASR use becomes more widespread. We are releasing
contextual biasing lists to accompany the Earnings21 dataset, creating a public
benchmark for this task. We present baseline results on this benchmark using a
pretrained end-to-end ASR model from the WeNet toolkit. We show results for
shallow fusion contextual biasing applied to two different decoding algorithms.
Our baseline results confirm observations that end-to-end models struggle in
particular with words that are rarely or never seen during training, and that
existing shallow fusion techniques do not adequately address this problem. We
propose an alternate spelling prediction model that improves recall of rare
words by 34.7% relative and of out-of-vocabulary words by 97.2% relative,
compared to contextual biasing without alternate spellings. This model is
conceptually similar to ones used in prior work, but is simpler to implement as
it does not rely on either a pronunciation dictionary or an existing
text-to-speech system.

本文介绍了一个使用上下文偏置列表对 ASR 进行计算的公共基准任务，并提出了一种替代拼写预测模型，它相对于不使用替代拼写的上下文偏置提高了罕见单词的召回率和词汇外单词的召回率。