Knowledge distillation(KD) is a common approach to improve model performance
in automatic speech recognition (ASR), where a student model is trained to
imitate the output behaviour of a teacher model. However, traditional KD
methods suffer from teacher label storage issue, especially when the training
corpora are large. Although on-the-fly teacher label generation tackles this
issue, the training speed is significantly slower as the teacher model has to
be evaluated every batch. In this paper, we reformulate the generation of
teacher label as a codec problem. We propose a novel Multi-codebook Vector
Quantization (MVQ) approach that compresses teacher embeddings to codebook
indexes (CI). Based on this, a KD training framework (MVQ-KD) is proposed where
a student model predicts the CI generated from the embeddings of a
self-supervised pre-trained teacher model. Experiments on the LibriSpeech
clean-100 hour show that MVQ-KD framework achieves comparable performance as
traditional KD methods (l1, l2), while requiring 256 times less storage. When
the full LibriSpeech dataset is used, MVQ-KD framework results in 13.8% and
8.2% relative word error rate reductions (WERRs) for non -streaming transducer
on test-clean and test-other and 4.0% and 4.9% for streaming transducer. The
implementation of this work is already released as a part of the open-source
project icefall.

本文提出了一种新的多码本量化方法，将生成教师向量卡尺化成为一个编解码问题，实现了教师模型的向量压缩，以减少模型存储需求。在 LibriSpeech 数据集上，用该方法训练的学生模型能够获得与传统 KD 方法（l1, l2）相当的 ASR 性能，并在存储容量上实现了 256 倍的减少。