Recently, pioneer work finds that speech pre-trained models can solve
full-stack speech processing tasks, because the model utilizes bottom layers to
learn speaker-related information and top layers to encode content-related
information. Since the network capacity is limited, we believe the speech
recognition performance could be further improved if the model is dedicated to
audio content information learning. To this end, we propose Intermediate Layer
Supervision for Self-Supervised Learning (ILS-SSL), which forces the model to
concentrate on content information as much as possible by adding an additional
SSL loss on the intermediate layers. Experiments on LibriSpeech test-other set
show that our method outperforms HuBERT significantly, which achieves a
23.5%/11.6% relative word error rate reduction in the w/o language model
setting for base/large models. Detailed analysis shows the bottom layers of our
model have a better correlation with phonetic units, which is consistent with
our intuition and explains the success of our method for ASR.

利用中间层监督自监督学习（ILS-SSL）对语音预训练模型进行优化，可以更好地集中于音频内容信息学习，从而实现识别性能的提高，并且在语言模型不被使用情况下，相对字错率下降 23.5%。