This study presents a novel approach for knowledge distillation (KD) from a BERT teacher model to an automatic speech recognition (ASR) model using intermediate layers. To distil the teacher's knowledge, we use an attention decoder that learns from BERT's token probabilities. Our method shows that language model (LM) information can be more effectively distilled into an ASR model using both the intermediate layers and the final layer. By using the intermediate layers as distillation target, we can more effectively distil LM knowledge into the lower network layers. Using our method, we achieve better recognition accuracy than with shallow fusion of an external LM, allowing us to maintain fast parallel decoding. Experiments on the LibriSpeech dataset demonstrate the effectiveness of our approach in enhancing greedy decoding with connectionist temporal classification (CTC).

通过使用中间层和最终层，本研究提出了一种将BERT教师模型的知识蒸馏到自动语音识别模型中的新方法。实验证明，使用中间层作为蒸馏目标可以更有效地将语言模型知识蒸馏到较低的网络层，从而实现比外部语言模型的浅层融合更好的识别准确性，同时保持了快速并行解码。

将解码与知识蒸馏并行进行：从语言模型到端到端语音识别的有效知识蒸馏