We propose a novel deliberation-based approach to end-to-end (E2E) spoken
language understanding (SLU), where a streaming automatic speech recognition
(ASR) model produces the first-pass hypothesis and a second-pass natural
language understanding (NLU) component generates the semantic parse by
conditioning on both ASR's text and audio embeddings. By formulating E2E SLU as
a generalized decoder, our system is able to support complex compositional
semantic structures. Furthermore, the sharing of parameters between ASR and NLU
makes the system especially suitable for resource-constrained (on-device)
environments; our proposed approach consistently outperforms strong pipeline
NLU baselines by 0.60% to 0.65% on the spoken version of the TOPv2 dataset
(STOP). We demonstrate that the fusion of text and audio features, coupled with
the system's ability to rewrite the first-pass hypothesis, makes our approach
more robust to ASR errors. Finally, we show that our approach can significantly
reduce the degradation when moving from natural speech to synthetic speech
training, but more work is required to make text-to-speech (TTS) a viable
solution for scaling up E2E SLU.

本文提出了一种新颖的基于推理的端到端（E2E）口语语音理解（SLU）方法，通过在自动语音识别（ASR）的文本和音频嵌入上进行条件控制，流式传输的 ASR 模型产生第一步的假设并由第二步的自然语言理解（NLU）组件生成语义解析。