Large-scale pre-trained language models (PLMs) with powerful language
modeling capabilities have been widely used in natural language processing. For
automatic speech recognition (ASR), leveraging PLMs to improve performance has
also become a promising research trend. However, most previous works may suffer
from the inflexible sizes and structures of PLMs, along with the insufficient
utilization of the knowledge in PLMs. To alleviate these problems, we propose
the hierarchical knowledge distillation on the continuous integrate-and-fire
(CIF) based ASR models. Specifically, we distill the knowledge from PLMs to the
ASR model by applying cross-modal distillation with contrastive loss at the
acoustic level and applying distillation with regression loss at the linguistic
level. On the AISHELL-1 dataset, our method achieves 15% relative error rate
reduction over the original CIF-based model and achieves comparable performance
(3.8%/4.1% on dev/test) to the state-of-the-art model.

该论文研究了如何利用预训练语言模型来提高自动语音识别的性能，提出了基于分层知识蒸馏的方法，并使用交叉模态蒸馏和对比损失以及回归损失来跨语言层面和声学层面校准自动语音识别模型，实现了相对误差率降低 15％，并且达到了同类最先进模型的性能。