This study presents a novel approach for knowledge distillation (KD) from a
BERT teacher model to an automatic speech recognition (ASR) model using
intermediate layers. To distil the teacher's knowledge, we use an attention
decoder that learns from BERT's token probabilities. Our method shows that
language model (LM) information can be more effectively distilled into an ASR
model using both the intermediate layers and the final layer. By using the
intermediate layers as distillation target, we can more effectively distil LM
knowledge into the lower network layers. Using our method, we achieve better
recognition accuracy than with shallow fusion of an external LM, allowing us to
maintain fast parallel decoding. Experiments on the LibriSpeech dataset
demonstrate the effectiveness of our approach in enhancing greedy decoding with
connectionist temporal classification (CTC).

通过使用中间层和最终层，本研究提出了一种将 BERT 教师模型的知识蒸馏到自动语音识别模型中的新方法。实验证明，使用中间层作为蒸馏目标可以更有效地将语言模型知识蒸馏到较低的网络层，从而实现比外部语言模型的浅层融合更好的识别准确性，同时保持了快速并行解码。

将解码与知识蒸馏并行进行：从语言模型到端到端语音识别的有效知识蒸馏

Keep Decoding Parallel with Effective Knowledge Distillation from  Language Models to End-to-end Speech Recognisers

The Conformer model is an excellent architecture for speech recognition
modeling that effectively utilizes the hybrid losses of connectionist temporal
classification (CTC) and attention to train model parameters. To improve the
decoding efficiency of Conformer, we propose a novel connectionist temporal
summarization (CTS) method that reduces the number of frames required for the
attention decoder fed from the acoustic sequences generated by the encoder,
thus reducing operations. However, to achieve such decoding improvements, we
must fine-tune model parameters, as cross-attention observations are changed
and thus require corresponding refinements. Our final experiments show that,
with a beamwidth of 4, the LibriSpeech's decoding budget can be reduced by up
to 20% and for FluentSpeech data it can be reduced by 11%, without losing ASR
accuracy. An improvement in accuracy is even found for the LibriSpeech
"test-other" set. The word error rate (WER) is reduced by 6\% relative at the
beam width of 1 and by 3% relative at the beam width of 4.

本研究提出了一种新的” 连接时序总结 “(CTS) 方法，它可以减少 Conformer 模型中 attention decoder 所需的 frames 数，提高解码效率，并且它能够在不损失 ASR 准确性的情况下降低解码预算，提高识别准确度.