In this paper, we provide a new perspective on self-supervised speech models
from how the self-training targets are obtained. We generalize the targets
extractor into Offline Targets Extractor (Off-TE) and Online Targets Extractor
(On-TE). Based on this, we propose a new multi-tasking learning framework for
self-supervised learning, MT4SSL, which stands for Boosting Self-Supervised
Speech Representation Learning by Integrating Multiple Targets. MT4SSL refers
to two typical models, HuBERT and data2vec, which use the K-means algorithm as
an Off-TE and a teacher network without gradients as an On-TE, respectively.
Our model outperforms previous SSL methods by nontrivial margins on the
LibriSpeech benchmark, and is comparable to or even better than the
best-performing models with no need for that much data. Furthermore, we find
that using both Off-TE and On-TE results in better convergence in the
pre-training phase. With both effectiveness and efficiency, we think that doing
multi-task learning on self-supervised speech models from our perspective is a
promising trend.

本文从获取自我训练目标的角度提供了一个新的自监督语音模型视角，并将目标提取器概括为离线目标提取器（Off-TE）和在线目标提取器（On-TE），在此基础上，提出了一个新的多任务学习框架 MT4SSL（即将多个目标整合以提高自监督语音表示学习），该模型在 LibriSpeech 基准测试中优于以前的 SSL 方法，并且在不需要超过数据的情况下，可与最佳表现模型相媲美甚至更好，在预训练阶段中使用 Off-TE 和 On-TE 可导致更好的收敛，因此我们认为从我们的角度在自监督语音模型上进行多任务学习是一个有前途的趋势。