There has been an increased interest in the integration of pretrained speech
recognition (ASR) and language models (LM) into the SLU framework. However,
prior methods often struggle with a vocabulary mismatch between pretrained
models, and LM cannot be directly utilized as they diverge from its NLU
formulation. In this study, we propose a three-pass end-to-end (E2E) SLU system
that effectively integrates ASR and LM subnetworks into the SLU formulation for
sequence generation tasks. In the first pass, our architecture predicts ASR
transcripts using the ASR subnetwork. This is followed by the LM subnetwork,
which makes an initial SLU prediction. Finally, in the third pass, the
deliberation subnetwork conditions on representations from the ASR and LM
subnetworks to make the final prediction. Our proposed three-pass SLU system
shows improved performance over cascaded and E2E SLU models on two benchmark
SLU datasets, SLURP and SLUE, especially on acoustically challenging
utterances.

我们提出了一个三步走的端到端 SLU 系统，将预训练的语音识别模型和语言模型有效地集成到 SLU 框架中，以用于序列生成任务。该系统在 ASR 和 LM 子网络的基础上进行预测，最终预测基于 ASR 和 LM 子网络的表示结果。我们的三步走 SLU 系统在两个基准 SLU 数据集（SLURP 和 SLUE）上表现出卓越性能，尤其在具有挑战性的音频对话中的表现更好。