The ability to learn new concepts sequentially is a major weakness for modern neural networks, which hinders their use in non-stationary environments. Their propensity to fit the current data distribution to the detriment of the past acquired knowledge leads to the catastrophic forgetting issue. In this work we tackle the problem of Spoken Language Understanding applied to a continual learning setting. We first define a class-incremental scenario for the SLURP dataset. Then, we propose three knowledge distillation (KD) approaches to mitigate forgetting for a sequence-to-sequence transformer model: the first KD method is applied to the encoder output (audio-KD), and the other two work on the decoder output, either directly on the token-level (tok-KD) or on the sequence-level (seq-KD) distributions. We show that the seq-KD substantially improves all the performance metrics, and its combination with the audio-KD further decreases the average WER and enhances the entity prediction metric.

该研究提出了三种知识蒸馏方法，通过应用于声音编码器(audio-KD)和解码器(tok-KD和seq-KD)以解决神经网络在非稳态环境下学习新概念的问题，特别应用于连续学习等领域的口语理解问题。结果表明，seq-KD可以显著提高性能，并与audio-KD的结合进一步降低了平均WER并增强了实体预测指标。

端到端逐步增量式语音理解的序列级知识蒸馏