Spoken language understanding (SLU) systems extract transcriptions, as well as semantics of intent or named entities from speech, and are essential components of voice activated systems. SLU models, which either directly extract semantics from audio or are composed of pipelined automatic speech recognition (ASR) and natural language understanding (NLU) models, are typically trained via differentiable cross-entropy losses, even when the relevant performance metrics of interest are word or semantic error rates. In this work, we propose non-differentiable sequence losses based on SLU metrics as a proxy for semantic error and use the REINFORCE trick to train ASR and SLU models with this loss. We show that custom sequence loss training is the state-of-the-art on open SLU datasets and leads to 6% relative improvement in both ASR and NLU performance metrics on large proprietary datasets. We also demonstrate how the semantic sequence loss training paradigm can be used to update ASR and SLU models without transcripts, using semantic feedback alone.

本文提出了基于SLU度量标准的非可区分序列损失作为语义误差的代理，并使用REINFORCE技巧训练ASR和SLU模型。作者展示了自定义序列损失训练在开放SLU数据集上的最新技术水平，并在大型专有数据集上显示了6％的相对改进，同时提高了ASR和NLU性能指标。作者还展示了如何使用语义反馈单独更新ASR和SLU模型。

以意传意，不在于言辞：面向口语理解的序列损失训练