Continued self-supervised (SSL) pre-training for adapting existing SSL models
to the target domain has shown to be extremely effective for low-resource
Automatic Speech Recognition (ASR). This paper proposes Stable Distillation, a
simple and novel approach for SSL-based continued pre-training that boosts ASR
performance in the target domain where both labeled and unlabeled data are
limited. Stable Distillation employs self-distillation as regularization for
continued pre-training, alleviating the over-fitting issue, a common problem
continued pre-training faces when the source and target domains differ.
Specifically, first, we perform vanilla continued pre-training on an initial
SSL pre-trained model on the target domain ASR dataset and call it the teacher.
Next, we take the same initial pre-trained model as a student to perform
continued pre-training while enforcing its hidden representations to be close
to that of the teacher (via MSE loss). This student is then used for downstream
ASR fine-tuning on the target dataset. In practice, Stable Distillation
outperforms all our baselines by 0.8 - 7 WER when evaluated in various
experimental settings.

通过自监督预训练（SSL）持续适应现有的 SSL 模型至目标领域已被证明对于资源稀缺的自动语音识别（ASR）非常有效。本文提出了一种简单而新颖的方法，称为稳定蒸馏，用于基于 SSL 的持续预训练，促进目标领域 ASR 性能的提升，其中有限的有标签和无标签数据。稳定蒸馏将自蒸馏作为持续预训练的正则化方式，减轻持续预训练在源领域与目标领域不同时所面临的过拟合问题。具体而言，首先，在目标领域 ASR 数据集上对初始 SSL 预训练模型执行普通的持续预训练，并将其称为教师。接下来，将相同的初始预训练模型作为学生，执行持续预训练，同时强制其隐藏表示与教师的表示接近（通过 MSE 损失）。然后，使用该学生在目标数据集上进行下游 ASR 微调。在实践中，当在不同的实验设置中评估时，稳定蒸馏胜过了我们的所有基线方法，WER 提高 0.8-7 个百分点。