In recent years, self-supervised pre-training methods have gained significant
traction in learning high-level information from raw speech. Among these
methods, HuBERT has demonstrated SOTA performance in automatic speech
recognition (ASR). However, HuBERT's performance lags behind data2vec due to
disparities in pre-training strategies. In this paper, we propose (i) a Swap
method to address pre-training and inference mismatch observed in HuBERT and
(ii) incorporates Multicluster masked prediction loss for more effective
utilization of the models capacity. The resulting method is, MS-HuBERT, an
end-to-end self-supervised pre-training method for learning robust speech
representations. It beats vanilla HuBERT on the ASR Librispeech benchmark on
average by a 5% margin when evaluated on different finetuning splits.
Additionally, we demonstrate that the learned embeddings obtained during
pre-training encode essential information for improving performance of content
based tasks such as ASR.

提出了一种用于学习鲁棒语音表示的自我监督预训练方法 MS-HuBERT，通过解决预训练和推理不匹配问题，以及利用模型能力更有效地使用多聚类遮蔽预测损失，相比于传统的 HuBERT 在不同微调数据集上平均提高了 5% 的性能，并表明在预训练期间获得的嵌入式表示编码了提高内容相关任务（如 ASR）性能的重要信息。