Recently, end-to-end automatic speech recognition models based on
connectionist temporal classification (CTC) have achieved impressive results,
especially when fine-tuned from wav2vec2.0 models. Due to the conditional
independence assumption, CTC-based models are always weaker than
attention-based encoder-decoder models and require the assistance of external
language models (LMs). To solve this issue, we propose two knowledge
transferring methods that leverage pre-trained LMs, such as BERT and GPT2, to
improve CTC-based models. The first method is based on representation learning,
in which the CTC-based models use the representation produced by BERT as an
auxiliary learning target. The second method is based on joint classification
learning, which combines GPT2 for text modeling with a hybrid CTC/attention
architecture. Experiment on AISHELL-1 corpus yields a character error rate
(CER) of 4.2% on the test set. When compared to the vanilla CTC-based models
fine-tuned from the wav2vec2.0 models, our knowledge transferring method
reduces CER by 16.1% relatively without external LMs.

基于 CTC 的自动语音识别模型一直比基于处理单元的编码 - 解码模型弱，本文提出了两种知识转移方法，借助预训练的 BERT 和 GPT2 来提高 CTC-based 模型的性能。在实验中，相对于不使用外部语言模型的基础模型，我们的方法将字符错误率降低了 16.1%。