End-to-end (E2E) models are becoming increasingly popular for spoken language
understanding (SLU) systems and are beginning to achieve competitive
performance to pipeline-based approaches. However, recent work has shown that
these models struggle to generalize to new phrasings for the same intent
indicating that models cannot understand the semantic content of the given
utterance. In this work, we incorporated language models pre-trained on
unlabeled text data inside E2E-SLU frameworks to build strong semantic
representations. Incorporating both semantic and acoustic information can
increase the inference time, leading to high latency when deployed for
applications like voice assistants. We developed a 2-pass SLU system that makes
low latency prediction using acoustic information from the few seconds of the
audio in the first pass and makes higher quality prediction in the second pass
by combining semantic and acoustic representations. We take inspiration from
prior work on 2-pass end-to-end speech recognition systems that attends on both
audio and first-pass hypothesis using a deliberation network. The proposed
2-pass SLU system outperforms the acoustic-based SLU model on the Fluent Speech
Commands Challenge Set and SLURP dataset and reduces latency, thus improving
user experience. Our code and models are publicly available as part of the
ESPnet-SLU toolkit.

本研究利用内部预训练的语言模型构建强大的语义表示，使用 2 段 SLU 系统，结合语义和语音特征进行高质量语音识别，以提高用户体验和降低延迟。