Continued pre-training (CP) offers multiple advantages, like target domain
adaptation and the potential to exploit the continuous stream of unlabeled data
available online. However, continued pre-training on out-of-domain
distributions often leads to catastrophic forgetting of previously acquired
knowledge, leading to sub-optimal ASR performance. This paper presents FusDom,
a simple and novel methodology for SSL-based continued pre-training. FusDom
learns speech representations that are robust and adaptive yet not forgetful of
concepts seen in the past. Instead of solving the SSL pre-text task on the
output representations of a single model, FusDom leverages two identical
pre-trained SSL models, a teacher and a student, with a modified pre-training
head to solve the CP SSL pre-text task. This head employs a cross-attention
mechanism between the representations of both models while only the student
receives gradient updates and the teacher does not. Finally, the student is
fine-tuned for ASR. In practice, FusDom outperforms all our baselines across
settings significantly, with WER improvements in the range of 0.2 WER - 7.3 WER
in the target domain while retaining the performance in the earlier domain.

本文介绍了一种名为 FusDom 的简单而创新的基于自监督学习的连续预训练方法，通过在两个具有修改的预训练头部的相同自监督学习模型之间使用交叉注意机制，实现有弹性且不遗忘过去所见概念的语音表示学习，从而改进了在目标领域的自动语音识别性能。